World Models for Learning Dexterous Hand-Object Interactions from Human Videos

1 Introduction

• DexWM targets a gap left by world models whose text, navigation, or whole-body action representations are too coarse for dexterous dynamics.Dexterous manipulation also lacks large-scale robot datasets, motivating training from human data.Source: §1 · p. 2
• DexWM predicts future latent states from past states, dexterous hand actions, and camera motions using human and non-dexterous robot data.Actions are represented as differences in 3D hand keypoints and camera poses, enabling cross-embodiment dynamics learning.Source: §1 · p. 2
• An auxiliary hand-consistency objective jointly trains environment-state and hand-configuration prediction because latent-state prediction alone poorly simulates hand locations.This richer learning signal improves modeling of fine-grained hand dynamics.Source: §1 · p. 2
• DexWM achieves an 83% success rate on real-world object grasping with a Franka Panda robot and Allegro grippers.The system uses MPC planning and approximately four hours of exploratory, non-task-specific dexterous simulation data for embodiment adaptation.Source: §1 · p. 2
• DexWM supports zero-shot transfer to unseen grasping, placing, and reaching tasks through test-time MPC action optimization.Unlike behavior-cloning methods that directly predict actions or waypoints, it plans waypoint trajectories with a state-transition model.Source: §1 · p. 2

• Diffusion and Flow Matching models generate realistic videos, but autoregressive generation can accumulate errors over time during interactive streaming.Source: §2 · pp. 2–3
• Dexterous dynamics remain more challenging than navigation or full-body control because hand–object interactions require precise, fine-grained control.Source: §2 · p. 3
• Robotics world models support reinforcement learning and policy optimization, while dexterous applications include particle dynamics, articulated-object modeling, and scene-action-conditioned video diffusion.Source: §2 · p. 3
• Dexterous manipulation research has shifted from handcrafted gripper gaiting toward learning-based and sim2real methods, while scarce dexterous datasets motivate using egocentric human videos.Source: §2 · p. 3
• DexWM encodes images with a frozen DINOv2 encoder and predicts next latent states from states, hand actions, and camera motions for image and hand-keypoint reconstruction.Source: Figure 2 · p. 3

3 Proposed Methodology

• DexWM predicts future latent environment states from observed state histories and dexterous actions, then uses the model for manipulation planning.The latent state is encoded from egocentric RGB observations, while the predictor maps current or historical states and actions to future states.Source: §3 · pp. 3–4

3.1 State and Action Representation

• DINOv2 patch features represent latent states, emphasizing semantically meaningful scene structure rather than pixel-level appearance.Each latent state consists of patch-level features with P patches and feature dimension d.Source: §3.1 · p. 4
• DexWM represents dexterous actions as 3D hand-keypoint differences augmented with camera translation and orientation changes.The representation uses 21 keypoints per hand, transforms keypoints into a common coordinate frame, and includes camera motion to capture agent movement.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Predictor

• The predictor uses a Conditional Diffusion Transformer to condition future-state regression on action vectors and directly predicts DINOv2 features without iterative denoising.It consumes state history and the current action, assumes deterministic dynamics, and can autoregressively generate multistep predictions.Source: §3.2 · pp. 4–5

3.3 Hand Consistency Training Loss

• Because state-feature loss alone misses small hand regions, DexWM adds a hand-consistency loss that predicts fingertip and wrist heatmaps.Training combines latent-state and heatmap losses, with λ = 100 reported as best in the experiments.Source: §3.3 · p. 5

3.4 Robot Task Planning

• For robot transfer, DexWM optimizes joint-angle trajectories with CEM under a goal-conditioned planning objective.The planning cost combines latent-state distance with keypoint-location distance; this combination improves planning over latent-state cost alone, and grasping additionally uses an end-effector-orientation cost.Source: §3.4 · p. 5
• Training DexWM on EgoDex human videos in addition to DROID contributes to RoboCasa downstream open-loop performance, measured by lower embedding loss and higher PCK@20.Source: Table 1 · p. 6

4.2 Ablation Studies

• Adding EgoDex to DROID significantly improves RoboCasa performance while preserving EgoDex performance, supporting cross-embodiment benefits from human video.The ablation evaluates embedding L2 loss and PCK@20 on RoboCasa and EgoDex.Source: §4.2 · p. 6
• Embedding prediction error and keypoint overlap consistently improve as DexWM grows from 30M to 450M parameters.DexWM-XL is used by default, and blue versus red markers distinguish EgoDex+DROID from EgoDex-only training.Source: §4.2 · p. 6, Figure 5 · p. 6
• DexWM performs well across DINOv2, DINOv3, Web-SSL, V-JEPA 2, and SigLIP 2 backbones, with DINOv2 strongest overall and task-dependent variation.Some backbones may require additional tuning such as embedding normalization.Source: Figure 6 · p. 6, §4.2 · p. 7
• Adding the hand consistency loss yields up to a 34% increase in PCK@20 at 4 seconds and also benefits robot planning.The loss supplements embedding prediction to preserve fine-grained hand information.Source: §4.2 · p. 7, Table 2 · p. 7

4.4 Open-Loop Trajectory Evaluation

• DexWM achieves over 5 points higher average PCK@20 than competing world models, despite PEVA* slightly outperforming it in Embedding L2 error.PCK@20 measures local hand-keypoint accuracy, whereas L2 error captures overall perceptual similarity.Source: §4.4 · pp. 7–8, Table 3 · p. 7
• DexWM accurately follows unseen atomic hand motions and models physical interaction, including moving a cup forward when the hand collides with it.The controllability test specifies hand motion of 1 cm per frame.Source: §4.4 · p. 8, Figure 9 · p. 8

4.5 Human Video To Robot Transfer

• DexWM consistently outperforms all baselines in simulation transfer, including 28% success on the difficult place task and 53% average success versus 4% for similarly pretrained Diffusion Policy.The comparison indicates gains beyond human-data pretraining alone; DexWM without human-video pretraining performs substantially worse.Source: §4.5 · p. 9, Table 4 · p. 9
• Without real-robot data finetuning, DexWM achieved 10 successes out of 12 trials (≈83% success rate), while Diffusion Policy trained on exploratory simulation data failed on the real task.The zero-shot evaluation deployed DexWM on a real-world Franka arm with an Allegro gripper.Source: §4.5 · p. 9

• The framework demonstrates image-based goals, while text-specified goals are described as an extension rather than an evaluated setting.Source: §5 · p. 10
• The current setup assumes static scenes without external agents, so dynamic interactive environments remain outside the demonstrated scope.The authors suggest adding stochasticity through an extra latent variable optimized at test time.Source: §5 · p. 10
• Approximately four hours of exploratory simulation data are used to mitigate the embodiment gap, and removing this dependency is left for future work.Source: §5 · p. 10

6 Conclusion

• DexWM combines keypoint-based actions, patch-level state embeddings, and a hand-consistency loss to model dexterous behavior from human videos.Source: §6 · p. 10
• The model captures precise interaction dynamics under dexterous actions and transfers from human-video pretraining to previously unseen robotic grasping, reaching, and placing tasks.Source: §6 · p. 10
• The authors frame dexterous modeling and world models for robotics as directions toward more generalizable robots performing increasingly complex tasks.Source: §6 · p. 10

Appendix

• The appendix documents the architecture, planning algorithm, training and dataset methods, baselines, and additional qualitative visualizations.Source: Appendix · p. 15

A.1 Encoder

• Patch-level DINOv2 features represent latent states, emphasizing semantic image structure rather than unnecessary pixel-level details.Source: §A.1 · p. 15

A.2 Predictor

• The predictor applies Conditional Diffusion Transformer blocks with AdaLN action conditioning, using DexWM-XL by default across experiments.Source: §A.2 · p. 15, Table 5 · p. 15

A.3 Keypoint Predictor

• The keypoint head transforms latent states into twelve heatmaps covering ten fingertips and two wrists for the hand-consistency objective.Source: §A.3 · p. 15

A.4 Decoder

• A separately trained ViT-L decoder reconstructs 224 × 392 images for visualizing latent predictions rather than training the world model.Source: §A.4 · p. 15

B Planning Optimization

• CEM samples joint-angle sequences, rolls them through the world model, updates on elite trajectories, and executes actions in a receding-horizon MPC loop.Source: B Planning Optimization · pp. 15–16
• Planning time ranges from 38 to 168 seconds with similar performance across tested settings; the default 168-second configuration favors maximum robustness.Source: B Planning Optimization · p. 16, Figure 11 · p. 16
• The planning distribution is initialized from the robot’s starting configuration, with broader arm exploration and finer Allegro-gripper control.Source: B Planning Optimization · p. 16

C.1 Training Details

• The training loss combines future-state prediction and hand consistency, with λ = 100 yielding the best results while the DINOv2 encoder remains frozen.Source: §C.1 · p. 16
• DexWM is pretrained on EgoDex and DROID using eight preceding frames sampled over a maximum four-second context window.Source: §C.1 · p. 16
• The predictor receives multiple context lengths and predicts subsequent states, providing richer training signals through multi-step prediction.Source: §C.1 · p. 16
• Fine-tuning on exploratory RoboCasa simulation data for 50 epochs improves performance when multi-step prediction is incorporated.Source: §C.1 · p. 16

C.2 Datasets and Robotic Benchmarks

• 829 hours of egocentric video from EgoDex provide 3D hand keypoints, camera poses, and multimodal annotations for dexterous manipulation.EgoDex contains 194 tasks involving 500 objects, with approximately 1% held out for testing.Source: §C.2 · p. 17
• About 100 hours of DROID robot data complement EgoDex with non-dexterous manipulation sequences and generated dummy hand keypoints.The combined data support training across human and robot embodiments.Source: §C.2 · pp. 16–17
• RoboCasa evaluates reach, grasp, and place tasks over 50 randomized trajectories per task after approximately four hours of exploratory fine-tuning data.The exploratory sequences use randomized objects and noisy replayed actions to provide broad environment coverage rather than task-specific skills.Source: §C.2 · p. 17
• The four-finger Allegro gripper is mapped to DexWM’s five-finger action space by duplicating the last Allegro finger’s keypoints for the pinky.Only the right hand is used, while left-hand actions are set to zero.Source: §C.2 · p. 17
• The real-world benchmark uses 12 grasping trajectories with four objects and varied configurations on the same Franka Panda–Allegro setup as simulation.Real-world success is assessed by manually observing whether the object is held in the gripper.Source: §C.2 · pp. 16–17

C.3 Baselines

• Baselines include text-conditioned Cosmos-Predict2, navigation-conditioned NWM*, upper-body-pose-conditioned PEVA*, and goal-conditioned Diffusion Policy.NWM* excludes hand and body motion, PEVA* excludes fingers, and Diffusion Policy predicts action chunks from current observations and image goals.Source: §C.3 · pp. 17–18

C.4 Success Criteria for Simulation Tasks

• Reach succeeds when average fingertip-and-wrist distance from the goal is below 15 cm for 10 consecutive time steps.The threshold is evaluated in the frame reached by the robot against the goal frame.Source: §C.4 · p. 18
• Grasp succeeds when the wrist is within 20 cm of the object and gripper contact persists for 10 consecutive time steps.Both proximity and contact conditions must be satisfied.Source: §C.4 · p. 18
• Place succeeds when the manipulated object finishes within 10 cm of its goal-frame position for 10 consecutive time steps.The criterion measures final object-position agreement with the goal.Source: §C.4 · p. 18

C.5 Success Criteria for Real-World Grasping Tasks

• Real-world grasp success is determined by manual observation of whether the object is securely held in the gripper.Automatic distance and contact detection are not used for the real-world evaluation.Source: §C.5 · p. 18

C.6 Manipulation Failure Cases Breakdown

• Failure categories distinguish contact-dynamics failures, drops before the target, last-centimeter errors, and gradual latent-state drift.These categories identify whether failures arise during interaction, transport, final positioning, or state prediction.Source: §C.6 · pp. 18–19, Table 6 · p. 19

D Additional Visualizations

• Open-loop rollouts predict 20 frames at 5 Hz over a 4-second horizon and closely follow ground-truth trajectories in long-horizon dexterous tasks.Predicted latent states are decoded into images for visualization.Source: Figure 13 · p. 20
• Counterfactual-action visualizations show DexWM following different atomic right-hand actions while capturing corresponding environment dynamics.Examples include pulling a string upward and pushing a cup forward.Source: Figure 14 · p. 21
• Action-transfer visualizations show DexWM capturing fine-grained hand states that match a reference sequence in a new environment better than PEVA*.The comparison focuses on transferred hand configurations.Source: Figure 15 · p. 22
• Robot examples include zero-shot real-world grasping, simulation planning with MPC and the Cross-Entropy Method, and a collision-induced missed grasp.The simulation examples use no task-specific robot training, while one real-world failure occurs when grippers collide with an upside-down bowl.Source: D Additional Visualizations · p. 19, Figure 16 · p. 22, Figure 17 · p. 23