PaperlayerView exact revision on arXiv

Source-linked AI summary

Wh0: Generative World Models as Scalable Sources of Egocentric Human Hand Manipulation Data

Yangtao Chen, Zixuan Chen, Peiyang Wang, Yong-Lu Li, Jing Huo, Jieqi Shi, Yang Gao

arXiv:2606.22136v2cs.RO

TL;DR

Existing sources of dexterous-manipulation data trade off scale and alignment with robot deployment. Wh0 generates and converts egocentric human-hand videos into robot-trainable supervision, then co-trains with limited robot data; zero-shot success rises from 8.3% to 38.9% against robot-data-only post-training.

  • Problem

    Existing dexterous-manipulation data sources trade off scale and deployment alignment: teleoperation is expensive, simulation has a sim-to-real gap, and egocentric videos remain scene- and embodiment-misaligned.

    Source: §1 · p. 1

  • Method

    Wh0 generates a 50k-sample dataset of egocentric human-hand manipulation videos, instructions, and 3D hand-motion annotations, then co-trains it with limited real robot data.

    Source: §1 · p. 2

  • Results

    38.9% success with WM-H co-training versus 8.3% for VITRA post-trained only on robot data, a 4.7× gain.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Ablations show that scene alignment, embodiment alignment, and data scale contribute to gains, while WM-H mainly unlocks pretrained human-manipulation priors rather than learning dexterous skills from scratch.

    Source: §5.4 · p. 8, §6 · p. 9

    Wh0's experiments focus on single-arm pick-and-place; bimanual, tool-use, and longer-horizon tasks remain future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Scaling dexterous manipulation requires generalization across objects, scenes, and tasks, yet existing data sources face a trade-off between scale and scene/embodiment alignment: teleoperation data is well aligned with robot deployment but expensive to collect; simulation is scalable but limited by the sim-to-real gap; and real egocentric videos scale effectively but remain misaligned with robot deployment. We propose Wh0, a framework that uses generative video world models as scalable and controllable sources of egocentric human-hand manipulation data to unlock the manipulation capabilities of pretrained dexterous VLA models. Conditioned on language, objects, and scenes, Wh0 uses a generative world model to produce WM-H, a 50k-episode dataset of egocentric human-object interaction videos. Wh0 then converts the generated videos into robot-trainable supervision through hand motion reconstruction and visual editing. Co-trained with a limited amount of real robot data, WM-H adapts pretrained VLA models to dexterous manipulation deployment. Across 18 real-world dexterous manipulation tasks, compared with a model post-trained only on robot data, Wh0 improves zero-shot success on unseen tasks from 8.3% to 38.9%. Ablation studies further show that scalable generation and scene/embodiment alignment are key drivers of performance gains. Videos and open-source code can be found on our project website: https://chenyt31.github.io/wh0.github.io/.

1 Introduction

Wh0 frames generative video world models as a scalable source of egocentric human-hand manipulation data, addressing the scale and deployment-alignment trade-off across existing data sources. It builds WM-H and co-trains it with limited robot data; the resulting model improves zero-shot success on unseen tasks.

1 Introduction

  • Teleoperation is deployment-aligned but expensive, simulation scales but faces a sim-to-real gap, and real egocentric video scales but remains misaligned.VLA models leveraging egocentric video still face scene and embodiment gaps in dexterous manipulation.Source: §1 · p. 1
  • Generative world models serve as controllable, compute-driven sources of human-hand videos conditioned on language, objects, and scenes, rather than precise robot-dynamics simulators.WM-H is intended to combine scale with low scene and embodiment gaps to deployment.Source: §1 · p. 1, Figure 1 · p. 2
  • Wh0 generates WM-H, a 50k-sample dataset of egocentric manipulation videos, instructions, and 3D hand motion annotations, then co-trains it with limited real robot data.The approach treats scenes, objects, task types, and embodiment appearance as design variables that scale with compute rather than human labor.Source: §1 · p. 2
  • 38.9% success with WM-H co-training versus 8.3% for VITRA post-trained only on robot data, a 4.7× gain.Ablations identify scene alignment, embodiment alignment, and data scale as primary drivers of the gains.Source: §1 · pp. 2–3

2 Related Work

Prior work uses VLA models and egocentric human videos to support dexterous manipulation, while world models have served as dynamics models, visual predictors, and policy-learning substrates. Wh0 instead uses controllably generated human-hand videos as deployment-aligned post-training data alongside limited robot demonstrations.

2 Related Work

  • Prior VLA work uses egocentric videos, often with hand reconstruction, to learn manipulation priors, affordances, transferable hand actions, and policy representations, but passive videos are hard to control and remain deployment-misaligned.Other directions include hierarchical VLA frameworks and VLM-based planning systems.Source: §2 · p. 3
  • Robotics world models have served as dynamics models, visual predictors, and policy-learning substrates, while video-generation work converts synthesized interactions into robot supervision.Examples of extracted supervision include trajectories, point tracks, object poses, and 3D flows.Source: §2 · p. 3
  • Wh0 treats generated human-hand videos as post-training data co-trained with limited robot demonstrations to transfer manipulation capabilities from human-video pretraining.This differs from prior uses of world models as simulators, robot-trajectory generators, human-hand generators for retargeting or affordance inference, or policy backbones.Source: §2 · p. 3

3 WM-H Dataset Construction via Controllable Video Synthesis

WM-H is synthesized through controllable video generation, with instructions, scenes, and hand appearance designed for broad manipulation coverage and deployment alignment. The pipeline produces videos with language descriptions and reconstructed 3D hand poses as supervision.

WM-H Dataset Construction via Controllable Video Synthesis

  • WM-H contains 50k egocentric videos with 3D hand-pose annotations and language task descriptions, generated at approximately 5.44 GPU-hours per 1k videos.Its noun and adjective h-indexes are 201 and 117 across pick, place, and grasp instructions.Source: §3 · p. 3
  • A dual-agent instruction system expands object and attribute vocabularies, then samples under-represented words and filters duplicates for balanced coverage.Its templates assemble commands such as “pick the {adj} {noun}.”Source: §3 · p. 3
  • Scene-aligned backgrounds use the deployment camera viewpoint and a human-hand scale anchor; specified objects are edited into the scene before video synthesis.The video model animates the edited image conditioned on the instruction and an expected hand-object state-change description.Source: §3 · p. 4
  • Robot-hand editing replaces the human hand in selected frames while preserving pose, position, object motion, and scene composition.The pipeline renders the same manipulation trajectory with human- and robot-hand appearances.Source: §3 · p. 4
  • HaWoR reconstructs hand motion from generated human-hand videos, producing camera-space wrist poses and articulated MANO hand poses.MegaSAM camera tracking can associate per-frame predictions with the video camera trajectory.Source: §3 · pp. 4–5

4 Wh0: Policy Learning with Human-Robot Alignment

Wh0 co-finetunes a VITRA-style dexterous manipulation policy on generated WM-H data and limited teleoperated robot data. The policy predicts MANO-space hand motions, and the evaluation setup includes unseen objects and backgrounds.

4 Wh0: Policy Learning with Human-Robot Alignment

  • The VITRA-style policy conditions on PaliGemma features, FoV, and current hand state to denoise future actions in unified MANO space.Source: §4 · p. 5, Figure 3 · p. 5
  • Actions encode relative wrist translation and rotation plus 15-DoF MANO joint rotations; robot joints are retargeted to MANO using VITRA’s human-video normalization.Source: §4 · p. 5
  • The diffusion action decoder is trained with a noise-prediction MSE loss, using the current observation and language-conditioned features to predict future hand motions.The predicted actions are conditioned on the current hand state.Source: §4 · p. 5
  • The policy is initialized from VITRA, pretrained on Ego4D, Epic-Kitchens, EgoExo4D, and Something-Something-V2.Source: §4 · p. 5
  • The real-world setup uses a Unitree G1 with Inspire hands and evaluates seen and unseen objects across one seen and three unseen backgrounds; illustrated rollouts use unseen task combinations.Source: Figure 4 · p. 6, Figure 5 · p. 6
  • Each co-finetuning batch draws 28% teleoperation, 68% WM-H, and 4% WM-H EA from 50k WM-H samples and 400 teleoperated demonstrations.The mixture heavily oversamples robot data per sample.Source: Figure 3 · p. 5, §4 · p. 6

5 Experiments

The experiments evaluate Wh0’s zero-shot performance, the contribution of WM-H’s alignment and scale, and whether WM-H helps use pretrained human-manipulation priors.

5.1 Experimental Setup

  • The benchmark uses a Unitree G1 with Inspire hands and evaluates 18 tasks over four scenes, with 20 randomized trials per task and no task-specific demonstrations.The setup includes 400 expert trajectories collected on seen tasks and backgrounds.Source: §5.1 · pp. 6–7

5.2 Q1: Does Wh0 Improve Real-World Zero-Shot Dexterous Manipulation?

  • Co-training with additional human data substantially improves zero-shot success, and Wh0 rollouts appear more deployment-aligned than those from real egocentric videos.The comparison includes robot-data adaptation, human-video-pretrained VITRA, and a VITRA version co-trained with HOI4D videos.Source: §5.2 · p. 7

5.3 Q2: Which Properties of WM-H Matter?

  • Scene alignment keeps generated videos anchored to the target workspace, while embodiment alignment preserves pose and motion when editing frames to a robot hand.The ablation also examines grounding and task success across deployment alignment and WM-H scale; increasing WM-H scale improves both grounding metrics and real-world success.Source: Table 2 · p. 7, Figure 6 · p. 8, §5.3 · p. 8

5.4 Q3: Does WM-H Unlock Pretrained Human Manipulation Priors?

  • Human pretraining alone yields low Hand-Object Distance but zero task success; the strongest grounding and success come from combining human pretraining, teleoperation, and WM-H.Without human pretraining, teleoperation and WM-H still yield near-zero success, consistent with WM-H helping activate and align pretrained priors rather than teach dexterous skills from scratch.Source: §5.4 · p. 8, Table 3 · p. 8

6 Conclusion and Limitations

Wh0 uses world-model-generated human-hand videos and limited real robot demonstrations for VLA post-training, with reported gains in zero-shot dexterous manipulation. The paper also identifies limitations in generation, reconstruction, morphology, pretraining dependence, and task scope.

6 Conclusion and Limitations

  • Wh0 co-trains a 50k-sample dataset with limited real robot demonstrations, raising VITRA’s zero-shot success from 8.3% to 38.9% on a Unitree G1 with Inspire hands.The authors attribute gains to alignment and scale, and report that WM-H mainly unlocks human-video-pretrained manipulation priors rather than learning skills from scratch.Source: §6 · p. 9
  • Wh0 remains limited by generation quality, hand reconstruction accuracy, morphology mismatch, dependence on strong pretraining, and task scope.Experiments focus on single-arm pick-and-place; bimanual, tool-use, and longer-horizon tasks remain future work.Source: §6 · p. 9

A.1 Instruction Generation and Prompt Design

  • Instruction generation prioritizes low-frequency vocabulary and expands it only after existing words reach minimum usage thresholds, while rejecting duplicate instructions.The process tracks nouns, adjectives, instructions, and word frequencies, and filters newly generated words before adding them.Source: §A.1 · p. 16

A.2 Scene-Aligned Image Editing

  • Scene editing inserts required objects into randomly selected workspace images captured with the deployment camera, viewpoint, and resolution.A sampled guide region indicates the insertion area and is removed from the final image.Source: §A.2 · p. 16

A.3 Image-to-Video Generation

  • Image-to-video synthesis uses the edited scene as its initial frame and combines the task instruction with an inferred description of hand-object motion and temporal order.The video prompt specifies a fixed first-person top-down view and a single acting hand that stops after completing the action.Source: §A.3 · pp. 16–17

A.4 Embodiment-Aligned Image Editing

  • Embodiment editing replaces the human hand in sparsely sampled frames with a robot hand while keeping the original pose, position, scale, scene, and composition.The editing is restricted to hand appearance and material.Source: §A.4 · p. 17

A.5 Quality Evaluation

The quality evaluation reports practitioners’ realism judgments, ratings of synthetic-video quality, and ratings of hand alignment after editing.

A.5 Quality Evaluation

  • Across 355 trials, 134 (37.7%) AI-generated videos were perceived as real recordings, indicating substantial synthetic realism in scene layout, physical plausibility, and manipulation performance under higher-level evaluation criteria.Hand appearance and motion accounted for 34.9% of judgment cues, while low-level visual artifacts accounted for 19.4%.Source: §A.5 · p. 18
  • Synthetic-video ratings were 4.18 ± 1.09 for instruction alignment and 3.95 ± 1.19 for hand–object interaction on a 5-point scale.The study also reports object correctness at 3.97 ± 1.22, physical plausibility at 3.78 ± 1.30, and training suitability at 3.57 ± 1.31; the assumed real-video ceiling of 5.0 was not directly measured.Source: §A.5 · p. 18
  • Edited hands received pose-consistency ratings of 4.30 ± 0.85 and contact-preservation ratings of 4.25 ± 0.84; WM-H videos also show varied objects and manipulation tasks.The ratings indicate that edited hands largely preserve pre-edit pose and interactions.Source: §A.5 · pp. 17–18

A.6 Failure Cases

This appendix reports common WM-H failure modes and details the policy architecture, robot-to-human action alignment, and training and inference setup.

A.6 Failure Cases

  • WM-H failures include editing artifacts, physically implausible interactions, inconsistent transitions, invalid instructions, and unintended changes to scene objects during hand editing.The authors say these failure modes call for attention to physical plausibility, visual fidelity, consistency, instruction validity, and embodiment alignment.Source: §A.6 · pp. 18–19

B.1 Architecture and Conditioning.

  • The policy encodes observations and instructions with PaliGemma2-3B, then uses a cognition token’s final hidden state to condition a diffusion-based DiT-B action decoder.The action decoder predicts 16 future steps per chunk in a 102-dimensional MANO hand space.Source: §B.1 · p. 19

B.2 Human-Robot Action Alignment.

  • Robot states and actions are converted to the camera frame, retargeted to MANO, and normalized using statistics from human videos rather than robot-specific statistics.Edited WM-H frames with embodiment alignment provide additional training supervision.Source: §B.2 · p. 20

B.3 Optimization and Inference

  • Post-training freezes the vision encoder and updates the remaining backbone and diffusion decoder; inference uses 10-step DDIM sampling with classifier-free guidance scale 5.0.Training uses four H200 GPUs, a total batch size of 256, and a maximum of 40k steps.Source: §B.3 · p. 20

B.4 Data Mixture and Sampling Ratios

This subsection describes the training-data composition table and the episode annotation and balancing protocol used for the HOI4D baseline.

B.4 Data Mixture and Sampling Ratios

  • HOI4D is annotated by grouping every 100 frames into one episode with one language annotation, yielding 5,511 annotated episodes.Source: §B.4 · p. 20
  • The VITRA Real Version baseline is trained with a balanced mixture of teleoperated robot data and HOI4D episodes.Source: §B.4 · p. 20
  • Table B.4 summarizes training dataset compositions and sampling ratios, with R denoting teleoperated robot data, W denoting WM-H, and W-EA denoting embodiment-aligned WM-H.The supplied table caption identifies these dataset categories but does not report their individual values.Source: Table 5 · p. 20, §B.4 · p. 20

C.1 Real-World Experimental Setup

The evaluation uses a real humanoid robot and a benchmark of dexterous tasks, with stage-conditioned instructions for multi-phase tasks. It also describes a separate Hand-Object Distance evaluation as a proxy for instruction-conditioned object grounding.

C.1 Real-World Experimental Setup

  • The real-world benchmark uses a Unitree G1 with Inspire dexterous hands and an egocentric camera across 18 tasks, reporting mean success rate and standard deviation.The methods share the same hardware, viewpoint, and low-level control interface.Source: §C.1 · pp. 20–21
  • For multi-phase tasks, evaluators provide the instruction matching the current interaction stage and give the next instruction at each phase transition without manual correction.Source: §C.1 · p. 21
  • During pre-contact, a shared deployment rule constrains finger joints to move monotonically toward closure until a stable grasp is reached.The authors say this prior suppresses premature hand-opening motions near objects and improves real-robot stability.Source: §C.1 · p. 21

C.2 Hand-Object Distance Evaluation

  • Hand-Object Distance measures the gap between the predicted wrist action and estimated target-object position; lower distance indicates better instruction-conditioned grounding.It is a grounding proxy, not a complete task-success metric: low distance is usually necessary but not sufficient for successful manipulation.Source: §C.2 · p. 21

C.3 Robot Execution Results

Qualitative robot rollouts show Wh0 executing varied real-world manipulation actions and producing more consistent hand-object interactions and task alignment than the baseline VITRA policy.

C.3 Robot Execution Results

Related papers

Loading 2606.22136v2…