PaperlayerView exact revision on arXiv

Source-linked AI summary

LD4WAM: Learning Latent Dynamics from Human Videos for World Action Models

Zhenhao Shen, Jiaqi Liang, Jasper Lu, Feng Jiang, Yuran Wang, Chuanbo Wei, Jiayi Liu, Jianchun Yang, Qize Yu, Jiadi You, Ce Hao, Guanqi He, Chen Xie, Ruihai Wu

arXiv:2608.22403v1cs.RO

TL;DR

Human video offers scalable manipulation data, but pixel-prediction dynamics are not directly actionable and motion retargeting leaves embodiment gaps. LD4WAM learns motion-aligned semantic latent dynamics and transfers them from generated futures to action conditioning while preserving video generation. Across RoboTwin and two real-robot embodiments, it outperforms strong VLA and world action model baselines, with strong generalization to unseen objects and backgrounds.

  • Problem

    Human-video supervision in WAMs is often limited to pixel-level future-frame prediction, while motion retargeting leaves a visual embodiment gap for robot control.

  • Method

    LD4WAM learns motion-aligned latent dynamics with semantic reconstruction and real-motion alignment, then uses learnable queries in a Mixture-of-Transformers to condition action generation from predicted futures.

  • Results

    Across RoboTwin and two real-robot embodiments, LD4WAM outperforms strong VLA and world action model baselines and generalizes well to unseen objects and backgrounds.

  • Takeaways & Limitations

    Motion-aligned latent dynamics provide an embodiment-agnostic bridge from large-scale human and robot video priors to low-level manipulation actions.

  • Takeaways & Limitations

    Generalization remains weaker under unseen texture perturbations because action is conditioned on generated video, where texture prediction is difficult.

Abstract

from arXiv · show

Human video is playing an increasingly central role in training World Action Models (WAMs), owing to its diversity and low collection cost relative to teleoperated robot data. However, most WAMs learn from such video only by predicting pixel-level future frames, giving dynamics that are not directly actionable, whereas motion retargeting recovers directly actionable actions but leaves a large visual gap across embodiments. We therefore propose motion-aligned latent dynamics as an embodiment-agnostic representation to bridge video priors and low-level actions. We further present LD4WAM, which pairs a Latent Dynamics Model trained with semantic reconstruction and real motion alignment with a World Dynamics Action Model built as a mixture-of-transformers (MoT), which preserves full future-video generation and uses learnable queries to distill these latent dynamics from generated futures for action conditioning. Pretrained on our curated unified dataset of over 5{,}000 hours of human and robot data, LD4WAM performs strongly in RoboTwin simulation and on real robots equipped with both grippers and dexterous hands, while generalizing well to unseen objects and backgrounds.

Introduction

LD4WAM addresses the limited actionability and embodiment transfer of human-video dynamics by learning motion-aligned latent dynamics and using them to condition action generation while preserving future-video prediction. Pretrained on over 5,000 hours of human and robot data, it performs strongly across simulation and real-robot settings.

  • Motivation: Existing WAMs commonly learn from human video through pixel-level future-frame reconstruction, leaving learned dynamics weakly coupled to robot actions.This limits how effectively knowledge acquired from human video transfers to robot control.
  • Motivation: Motion retargeting produces actionable robot-space actions but can entangle physical knowledge with embodiment-specific execution and leave the visual embodiment gap unresolved.Alternative latent-action methods explain pixel changes but may retain appearance- and embodiment-specific variation.
  • Approach: Motion-aligned latent dynamics are compact inter-frame representations learned in semantic feature space and explicitly aligned with observed end-effector motion.Semantic abstraction suppresses appearance- and embodiment-specific details, while motion alignment grounds the representation in real motion.
  • Approach: The Latent Dynamics Model combines next-frame semantic reconstruction with motion alignment, while the World Dynamics Action Model uses learnable queries to distill latent dynamics from generated futures for action conditioning.The design preserves full future-video generation while providing the action expert with a compact dynamics signal.
  • Results: Over 5,000 hours of human and robot manipulation data support experiments in RoboTwin and on real robots with grippers and dexterous hands.The reported evaluation includes long-horizon, high-precision, and dexterous tasks, with generalization to unseen objects and backgrounds.
  • Contributions: The paper contributes motion-aligned latent dynamics, a query-mediated World Dynamics Action Model, and a unified large-scale cross-embodiment dataset.These contributions connect video priors with low-level action learning while retaining future-video generation.

Related Work

Prior work either retargets human motion into robot actions or learns transferable visual and dynamics representations from human video. LD4WAM instead grounds semantic latent dynamics in real motion while carrying them to the action expert.

  • Learning From Egocentric Human Videos: Human manipulation video is used either by retargeting demonstrations into a shared action space or by learning transferable visual or dynamics representations.The two lines differ in whether human data directly supplies action mappings or pretraining representations.
  • Learning From Egocentric Human Videos: Retargeting methods require tight viewpoint, speed, and kinematics alignment and can break down for grippers or dexterous hands because of the embodiment gap.They map human demonstrations into robot action spaces but remain sensitive to cross-embodiment correspondence.
  • Learning From Egocentric Human Videos: Representation-learning methods use human video for transferable visual or dynamics features, but leave the policy to recover the representation-to-action mapping from limited robot data.The cited limitation is that the representation is not grounded in real motion.
  • World Action Models: World action models unify future observation prediction and action generation, predominantly through pixel-space video generators or richer semantic and 3D prediction targets.Existing richer feature-space and 3D approaches still do not provide a representation that is both embodiment-independent and executable.
  • LD4WAM: LD4WAM follows the representation-learning line but grounds latent dynamics in real motion and transfers them to the action expert through learnable queries.This combines semantic feature-space learning with executable motion alignment.

Method

LD4WAM curates and standardizes human and robot manipulation data, learns motion-aligned latent dynamics, and integrates them into a staged Mixture-of-Transformers action model. Asymmetric attention makes the information flow video → latent dynamics → action while preserving video generation.

  • Data Processing: Human and robot datasets are unified in LeRobot format after subtask segmentation, duration filtering, 15 FPS downsampling, and human-video quality filtering.Action-annotated human clips are expressed in unified head-camera and wrist coordinate frames.
  • Data Processing: The resulting cross-embodiment corpus contains 274.66M frames, or 5,086 hours at 15 FPS, with 76.4% human and 23.6% robot data.The corpus aggregates eight datasets spanning egocentric human videos and multi-embodiment robot demonstrations.
  • Latent Dynamics Model: The LDM samples frames with random temporal skips, encodes them using frozen DINOv3, aggregates spatio-temporal features, and maps them through soft-VQ into latent-dynamics tokens.The representation is trained to capture changing content while discarding static appearance.
  • Latent Dynamics Model: The LDM loss combines semantic reconstruction, motion supervision from camera-frame end-effector deltas, and codebook regularization.Semantic dynamics use all video, while motion alignment additionally uses action-labeled frames; codebook terms prevent collapse.
  • World Dynamics Action Model: WDAM is a Mixture-of-Transformers with separate video, latent-dynamics, and action experts interacting through shared self-attention over concatenated tokens.The video expert generates future video and the action expert generates future action; the video expert is initialized from Wan2.2.
  • World Dynamics Action Model: Asymmetric attention forms a one-directional video → latent dynamics → action pipeline, with learnable queries distilling predicted video into a compact dynamics summary for control.The video stream remains self-contained, the query stream cannot see actions, and the action stream attends to both video and the summary.
  • Training Procedure: Training proceeds by pretraining the video and latent-dynamics experts on all data, aligning all experts on robot data, and post-training on task-specific data.The LDM is trained first and frozen; WDAM is then optimized across three stages with stage-specific loss weights.

Experiment

LD4WAM is evaluated across RoboTwin simulation, real-world gripper and dexterous-hand tasks, latent-dynamics retrieval, and generalization settings. It achieves strong task performance, cross-domain motion retrieval, and improved object and background robustness, while ablations identify motion alignment, semantic reconstruction, pretraining, and alignment training as important components.

  • Latent Dynamics Retrieval: Cross-domain nearest-neighbor retrieval consistently matches query motions across robot embodiments and between human hands and robot arms in the frozen LDM latent space.Opposite robot-arm queries retrieve correspondingly different motions, indicating motion-consistent retrieval across datasets.
  • Simulation: LD4WAM reaches 93.4% average success across 50 RoboTwin tasks, exceeding ACE-Ego-0 by +2.5 points and Lingbot-VA by +1.2 points.The top methods are tightly clustered in simulation, which the authors treat as a competitiveness check.
  • Real World: LD4WAM achieves the best success rate on six of seven real-world tasks, raising the average to 70.5% across gripper and dexterous-hand embodiments.The comparison averages are 63.3% for π0.5, 47.1% for Fast-WAM, and 64.3% for Lingbot-VA.
  • Generalization: On unseen objects, LD4WAM retains 88.6% of in-distribution performance, while background-shift gains over prior WAMs are +10.0 and +34.4 points.LD4WAM still trails π0.5 on background shifts, with the gap concentrated on texture-level perturbations because actions depend on generated future video.
  • Latent Dynamics Model Ablation: Removing motion alignment inflates latent-regression error by 3.7–5.0×, the largest LDM ablation effect, while hard quantization adds 76–88%.Semantic reconstruction preserves high-level content, and multi-stride sampling covers both large and small inter-frame motions.
  • Full Architecture Ablation: In the full architecture ablation, pretraining improves unseen-object and unseen-background performance by 7.8 and 23.3 points, while alignment middle-training adds 4.8 points overall.Together, these stages lift the baseline by 23.7 points; pretraining broadens visual and dynamic coverage, while alignment sharpens latent-dynamics use by the action expert.

Conclusion

LD4WAM uses motion-aligned latent dynamics to turn human and robot video into transferable manipulation capabilities. Across RoboTwin and two real-robot embodiments, it outperforms strong baselines and generalizes to unseen objects and backgrounds.

  • LD4WAM combines semantic reconstruction and real-motion alignment in an LDM with a WDAM that conditions action on distilled dynamics.
  • Across RoboTwin and two real-robot embodiments, LD4WAM outperforms strong VLA and world action model baselines.
  • LD4WAM shows strong generalization to unseen objects and backgrounds.

A. Data Processing Details

The data pipeline segments and filters manipulation videos, standardizes retained clips, and constructs unified coordinate frames for human and robot motion.

  • Source videos are segmented into subtask-level clips, with clips shorter than 3 s or longer than 60 s discarded and the remainder downsampled to 15 FPS.
  • The human wrist frame places its origin at the wrist, points z along the hand, defines x as the palmar normal, and sets y along the knuckle line.
  • Robot gripper frames place the origin at the arm attachment, point z along the gripper, and define x normal to jaw travel and y along jaw opening.
  • The exported motiond is the frame-to-frame difference of the constructed pose.
  • Action-annotated human clips are filtered for camera motion, missing hand annotations, and insufficient wrist motion before actions are expressed in unified camera and wrist frames.

B. Dataset Details

The pre-training corpus combines human and robot manipulation data from eight datasets in a unified format and sampling rate, with human data comprising most of the corpus.

  • The corpus contains five egocentric human datasets and three multi-embodiment robot datasets unified into LeRobot format at 15 FPS.
  • Human data forms the majority of the pre-training corpus.

C. Training Details

Training uses a compact LDM and a three-expert WDAM, with staged objectives and shared action modeling across gripper and dexterous-hand platforms.

  • The LDM uses a frozen DINOv3 encoder, a spatio-temporal transformer, and SoftVQ to produce a 512-dimensional latent vector per transition.
  • The corpus is re-filtered to roughly 1,500 hours for LDM training, with motion alignment applied to four datasets having verified end-effector delta labels.
  • WDAM is a three-expert Mixture-of-Transformers built on Wan2.2, with latent-dynamics and action experts alongside the video expert.
  • The Action Expert shares one transformer body across parallel-gripper and dexterous-hand robots, adapting through embodiment-specific input handling.
  • The real-robot platforms comprise a dual-arm AgileX PiPER with Pika Grippers and dual Tianji arms with Wuji hands.
  • Training follows stages that first learn video and latent dynamics, then align all experts on multi-view robot actions.

D. Real-World Setup and Task Description

The evaluation spans two real-robot embodiments and seven manipulation tasks covering gripper, bimanual, articulated, deformable, and dexterous skills.

  • Platforms: The hardware includes a dual-arm PiPER gripper platform and a dual-arm Tianji platform with Wuji dexterous hands.Both platforms use head and wrist cameras for observation.
  • Tasks: Seven tasks cover long-horizon sorting, precision insertion, articulated-object tidying, cloth folding, bimanual handover, dexterous placement, and in-hand tool use.Five tasks run on the gripper platform and two on the dexterous platform.
  • Tasks: The dexterous platform evaluates locating and placing a Rubik’s cube among distractors and operating a spray bottle through coordinated handoffs and trigger pressing.These tasks require multi-finger grasping and independent finger actuation.

E. Latent Dynamics Regression

The regression probe measures action-relevant information already encoded in fixed latent dynamics by predicting normalized end-effector motion with a shallow shared head across sampling rates.

  • Probe design: A frozen latent representation is evaluated by training only a shallow two-layer MLP to regress motion, isolating information present in the latent.The probe uses a deliberately shallow head to limit added capacity.
  • Data: The evaluation uses 2,500 clean and 2,500 randomized RoboTwin episodes across all 50 tasks, an unseen domain for the LDM.Near-static transitions are discarded, and clips are split 8:2 for probe training and validation.
  • Regression target: Each eight-frame clip produces seven transitions with a 12-D target containing both arms’ translation and rotation differences.Translation and rotation are scaled by 0.01 m and 0.08 rad, respectively.
  • Optimization: All rates use the same projection, MLP architecture, AdamW optimization, and 200-epoch cosine schedule, with a separate head per rate.The shared projection removes dimensionality differences between latent models.
  • Sampling rates: Six sampling rates span 50, 25, 12.5, 10, 5, and 2.5 Hz, probing small and large inter-frame motions.The rates form high-rate and low-rate bands derived from native 50 Hz rendering.

F. RoboTwin Detailed Results

LD4WAM achieves the highest average RoboTwin success under both clean and randomized evaluation, while real-world rollouts complete all tested generalization variations.

  • RoboTwin results: 93.96 clean / 92.78 randomized is the highest average success rate across all 50 RoboTwin tasks under both evaluation settings.Table 8 provides the per-task breakdown complementing these aggregate results.
  • Real-world generalization: Every Tidy Desk, Fold Shirt, and Handover Mug rollout succeeds across tested unseen-object and unseen-background variations.The variations include novel objects and scenes.
Loading 2608.22403v1…