PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniHumanoid: Streaming Cross-Embodiment Video Generation with Paired-Free Adaptation

Yiren Song, Xiyao Deng, Pei Yang, Yihan Wang, Mike Zheng Shou

arXiv:2605.12038v1cs.CV

TL;DR

Cross-embodiment video generation must preserve motion while rendering embodiment-specific appearance, yet existing methods struggle with scalability and practical deployment. OmniHumanoid factorizes motion transfer from embodiment adaptation and achieves strong motion fidelity, embodiment consistency, paired-free adaptation, and efficient long-horizon generation.

  • Problem

    Existing cross-embodiment video generators entangle motion with embodiment geometry and require costly paired data for each new robot, limiting scalability and deployment.

  • Method

    OmniHumanoid learns shared motion transfer from paired videos and adapts target embodiments with lightweight unpaired modules using isolated motion and appearance information flow.

  • Results

    Experiments on synthetic and real-world benchmarks show strong motion fidelity and embodiment consistency, paired-free adaptation to unseen humanoids, and efficient long-horizon generation.

  • Takeaways & Limitations

    The framework offers a scalable path to robot-centric video data generation by adapting unseen embodiments without retraining shared motion-transfer parameters.

  • Takeaways & Limitations

    Adaptation to a new embodiment assumes access only to unpaired videos and does not use paired source-target videos.

Abstract

from arXiv · show

Cross-embodiment video generation aims to transfer motions across different humanoid embodiments, such as human-to-robot and robot-to-robot, enabling scalable data generation for embodied intelligence. A major challenge in this setting is that motion dynamics are partly transferable across embodiments, whereas appearance and morphology remain embodiment-specific. Existing approaches often entangle these factors, and many require paired data for every target embodiment, which limits scalability to new robots. We present OmniHumanoid, a framework that factorizes transferable motion learning and embodiment-specific adaptation. Our method learns a shared motion transfer model from motion-aligned paired videos spanning multiple embodiments, while adapting to a new embodiment using only unpaired videos through lightweight embodiment-specific adapters. To reduce interference between motion transfer and embodiment adaptation, we further introduce a branch-isolated attention design that separates motion conditioning from embodiment-specific modulation. In addition, we construct a synthetic cross-embodiment dataset with motion-aligned paired videos rendered across diverse humanoid assets, scenes, and viewpoints. Experiments on both synthetic and real-world benchmarks show that OmniHumanoid achieves strong motion fidelity and embodiment consistency, while enabling scalable adaptation to unseen humanoid embodiments without retraining the shared motion model.

1 Introduction

OmniHumanoid addresses scalable cross-embodiment video generation by separating transferable motion from embodiment-specific appearance and adapting to unseen humanoids with unpaired videos. It further provides motion-aligned synthetic supervision and streaming generation for efficient, consistent robot video synthesis.

  • Introduction: Cross-embodiment video generation is difficult because motion, embodiment geometry, and kinematics are entangled, while paired data, identity preservation, temporal consistency, and interactive speed remain challenging.These obstacles limit scalable generation of demonstrations for new robot embodiments.
  • Introduction: OmniHumanoid factorizes generation into shared motion learning and embodiment-specific appearance modeling under the TAPE principle.The framework reuses motion knowledge across humanoids while preserving target-specific appearance and movement.
  • Introduction: Lightweight embodiment adapters learn robot-specific appearance and morphology from unpaired videos, while a unidirectional information flow isolates the shared motion pathway.The design prevents embodiment-specific priors from entering motion transfer and enables paired-free adaptation to unseen humanoids.
  • Introduction: OmniHumanoid introduces a motion-aligned synthetic dataset with shared motion and scene dynamics across diverse humanoids, scenes, cameras, and temporal structures.The dataset supports controlled motion-transfer supervision and evaluation on unseen robots, actions, and environments.
  • Introduction: 50 denoising steps are reduced to 4, enabling autoregressive cross-embodiment generation at 720p and 5 FPS on a single NVIDIA H200 GPU.A streaming video-to-video distillation pipeline converts an expensive bidirectional generator into an efficient causal generator.
  • Introduction: Experiments on synthetic and real-world benchmarks show strong motion fidelity, embodiment consistency, paired-free adaptation, and efficient long-horizon generation.The reported results support a scalable and practical path toward robot-centric video data generation.

2 Related Work

Related work centers on diffusion-based video generation, embodiment-gap reduction for embodied intelligence, and decoupled learning that separates distinct factors to improve controllability and generalization.

  • Diffusion-Based Generation: Diffusion models support image synthesis, image editing, video generation, and procedural generation, with DiT increasingly surpassing earlier GAN- and UNet-based methods.The passage attributes DiT’s advantage to enhanced visual fidelity and temporal consistency.
  • Embodiment Gap: Embodied-intelligence research addresses scarce robot image and video data by converting human-centric videos into robot-centric observations.First-person approaches jointly train on egocentric human and robot data, including EgoVLA, EgoMimic, and PH2D.
  • Decoupled Learning: Decoupled learning separates variation factors with different statistical structures to improve generative visual-model generalization and controllability.AnimateDiff separates spatial and temporal modeling, while MotionDirector disentangles motion and appearance using dual-path LoRA.

3 Methods

OmniHumanoid factorizes cross-embodiment generation into shared motion transfer and embodiment-specific adaptation, enabling unseen-embodiment learning from unpaired videos. Its branch-isolated architecture and causal distillation support motion-preserving synthesis and efficient long-horizon streaming.

  • Unseen-Embodiment Adaptation: For an unseen embodiment, OmniHumanoid trains a new LoRA from only tens of unpaired videos while keeping the shared motion transfer model frozen.Paired source–target videos are unavailable during this adaptation stage.
  • Factorized Architecture: The framework separates embodiment-invariant motion dynamics from embodiment-specific appearance and morphology using a shared motion transfer model and lightweight Embodiment Video LoRAs.The shared model encodes temporal dynamics, object interactions, and scene evolution, while each LoRA models geometry, texture, and structural priors.
  • Branch-Decoupled Architecture: An asymmetric attention mask lets the denoising branch read motion from the conditioning branch while isolating the conditioning branch from embodiment-specific updates.LoRA updates are applied only to the denoising branch, preserving transferable motion representations while rendering target-specific morphology and appearance.
  • Two-Stage Training: Two-stage training first pretrains each embodiment LoRA on unpaired videos, then trains shared motion transfer parameters on paired videos while freezing the LoRAs.A rolling LoRA loading strategy across target embodiments encourages the shared model to learn motion and scene-dynamics invariances.

4 Synthetic Cross-Embodiment Dataset

OmniHumanoid’s synthetic dataset uses motion-aligned rendering across diverse humanoid embodiments and environments to support factorized learning and cross-embodiment generalization. It provides paired supervision while reserving an unseen robot, motions, and scenes for evaluation.

  • Dataset construction: The Unity-based dataset draws over 700 humanoid motion sequences spanning manipulation, environment interaction, locomotion, and daily full-body activities.The sequences come from the Humoto motion library.
  • Motion-aligned rendering: Ten humanoid assets—five robots and five digital humans—share a unified skeleton topology and joint-level correspondence through Blender alignment and Unity retargeting.Compatible skeletons and animation controllers enable reliable cross-embodiment motion alignment.
  • Scene and embodiment variation: The dataset renders retargeted motions in 100 diverse office, factory, and outdoor scenes while varying only the humanoid asset within each scene.Paired videos preserve the same scene layout, camera viewpoint, and motion sequence, differing primarily in appearance and morphology.
  • Paired sample construction and evaluation split: Matched source–target embodiments yield 7,200 paired training samples across 100 scenes, while one robot is held out for testing with unseen motions and scene configurations.The held-out embodiment is excluded from all training stages to evaluate generalization.

5 Experiments

OmniHumanoid is evaluated on synthetic held-out and real-world benchmarks, combining reference-based image-quality metrics with vision-language assessment. Experiments show strong unseen-embodiment adaptation, while ablations identify motion-appearance decoupling as critical and streaming distillation as an efficiency-quality tradeoff.

  • Metrics: Synthetic evaluation reports PSNR, SSIM, and MSE, while both benchmarks additionally use Gemini-3 Flash for reference-free assessment.PSNR, SSIM, and MSE measure pixel-level reconstruction and motion-aligned visual fidelity on the synthetic benchmark.
  • Main Results: OmniHumanoid consistently outperforms V2V baselines, preserving motion trajectories while improving limb articulation and target-embodiment fidelity.Qualitative comparisons describe generic V2V models as suffering from motion drift, inconsistent limb articulation, and weak embodiment fidelity.
  • Ablation Studies: Removing motion-appearance decoupling drops the embodiment score from 8.43 to 2.53 and the motion score from 9.06 to 6.35.The results attribute these declines to interference between source-motion conditioning and target-embodiment rendering.
  • Ablation Studies: 4.85 FPS: the causal student accelerates generation from 0.10 FPS but reduces reconstruction and consistency quality relative to the bidirectional teacher.The full streaming student reaches 4.96 FPS and slightly recovers quality across SSIM, MSE, motion, embodiment, background, and overall scores.

6 Conclusion

OmniHumanoid decouples transferable motion learning from robot-specific appearance modeling for scalable human-to-robot video generation. It combines limited paired pretraining with lightweight paired-free adaptation to generalize to unseen robots without retraining the core model.

  • The framework separates transferable motion learning from robot-specific appearance modeling in human-to-robot video generation.
  • Paired pretraining on limited robot embodiments and RobotAppearance LoRA modules enable lightweight, paired-free adaptation to unseen robots without retraining the core model.
  • A decoupled attention architecture and rolling LoRA training strategy support the separation of motion learning and robot-specific appearance adaptation.

Appendix · A Prompt for Automated Evaluation

The appendix provides a comprehensive automated-evaluation prompt for generated videos, covering motion fidelity, embodiment similarity, background consistency, and overall quality. Each dimension specifies focused evaluation criteria, a 1–10 scoring rubric, and JSON-only output.

  • A Prompt for Automated Evaluation: Motion fidelity compares generated robot motion with the human condition video for movement accuracy, articulation, and temporal smoothness.The prompt examines pose, trajectory, speed, timing, joint movements, limb articulation, subtle actions, and unnatural jitter.
  • A Prompt for Automated Evaluation: The motion-fidelity evaluator uses a 1–10 scale ranging from perfect matching and fluidity to complete failure to follow the condition video.Scores 8–9 denote very accurate motion, while scores 1–3 indicate motion breakdown.
  • A Prompt for Automated Evaluation: Each evaluation prompt requires returning only a valid JSON object containing a numeric score and one short reason sentence.This output format is specified for motion fidelity, embodiment similarity, background consistency, and overall quality.
  • A Prompt for Automated Evaluation: Embodiment similarity compares the generated robot video with a reference robot image for identity, structural integrity, and temporal consistency.The criteria assess appearance, body proportions, mechanical features, colors, structural details, and morphing or identity changes across frames.
  • A Prompt for Automated Evaluation: Background consistency evaluates whether the generated video preserves a rigid, faithful, artifact-free background relative to the human condition video.It checks stability, detail and texture fidelity, flickering, texture crawling, warping near the robot, and hallucinated elements.
  • A Prompt for Automated Evaluation: Overall quality evaluates the generated robot video’s visual fidelity, realism, artifact absence, and holistic cinematic impression.The prompt considers sharpness, clarity, noise, blur, lighting, rendering, compositing, pixelation, smearing, and color shifts.

B User Study Results

A user study compared generated videos across Motion Fidelity, Embodiment Similarity, Background Consistency, and Overall Quality using participant preference votes. OmniHumanoid significantly outperformed existing baselines across all four metrics.

  • Study Design: Participants voted for the best method across Motion Fidelity, Embodiment Similarity, Background (BG) Consistency, and Overall Quality.Table 3 reports each method’s percentage of total user preference votes for the four metrics.
  • Results: OmniHumanoid significantly outperformed existing baselines across all four user-study metrics.The reported comparison covers Motion Fidelity, Embodiment Similarity, Background (BG) Consistency, and Overall Quality.
  • Results: Table 3 summarizes the percentage of total preference votes received by each method across the four evaluation metrics.The table presents user preferences rather than another quantitative evaluation measure.

C Limitations

OmniHumanoid improves generation efficiency but has limitations in few-step distillation. At extremely low inference steps, generated videos can lose fine-grained detail, temporal smoothness, and complex motion fidelity relative to the teacher model.

  • C Limitations: Few-step distillation causes noticeable performance degradation compared with the original teacher model.The limitation is observed specifically in the few-step distillation regime.
  • C Limitations: At extremely low inference steps, such as 4 steps, generated videos may compromise fine-grained details.These outputs are directly compared with high-quality teacher-generated videos.
  • C Limitations: Extremely low-step generation may also reduce temporal smoothness or complex motion fidelity.These compromises occur when outputs are directly compared with the high-quality outputs generated by the original teacher model.
Loading 2605.12038v1…