PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamX: Video Diffusion Model is a Universal Latent Action Learner

Jisoo Kim, Chaehyun Kim, Junsu Kim, Jiho Park, Seongchan Kim, Eunbeen Hong, Soowon Son, Fu-En Yang, Seungryong Kim

arXiv:2611.05377v1cs.CVcs.AIcs.RO

TL;DR

Human and robot action labels differ across embodiments, and video-derived latent actions can retain irrelevant visual information. DreamX extracts compact actions from a pretrained video diffusion model and uses its generative prior to supervise them. It reports state-of-the-art LARYBench performance and transfer across settings without paired training data.

  • Problem

    Human videos lack robot action labels and robot control spaces vary by embodiment, while reconstruction-trained latent actions can encode viewpoint and scene appearance alongside action.

    Source: §1 · p. 2

  • Method

    DreamX extracts compact latent actions from pretrained video-model spatiotemporal features and uses future-frame reconstruction with the model’s generative prior to supervise them.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    DreamX achieves state-of-the-art performance among latent action models on LARYBench and enables any-to-any video and human-to-robot action transfer without paired training data.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The shared latent actions also serve as prediction targets for policy learning from human and robot demonstrations.

    Source: §5 · p. 10

    Cross-embodiment transfer and robot policies may fail outside evaluated settings and require safety validation before deployment.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.

1 INTRODUCTION

Human and robot demonstrations use incompatible action labels, while video-derived latent actions can entangle motion with appearance. DreamX uses a pretrained video diffusion model to learn compact latent actions, reporting state-of-the-art LARYBench performance and transfer across settings.

1 INTRODUCTION

  • Human videos lack robot action labels, and different embodiments use different control spaces, so low-level action labels are difficult to share directly.A common representation must preserve shared motion and effects across embodiments.Source: §1 · p. 2
  • Latent action models compress visual transitions, but reconstruction objectives can leave their representations entangled with viewpoint and scene appearance.Separating action-relevant structure from domain-specific visual factors remains a challenge.Source: §1 · p. 2
  • Pretrained video diffusion models provide rich spatiotemporal representations and a generative prior for synthesizing future frames from an initial frame and additional conditioning.These properties motivate using their representations and a pretrained generative decoder for latent-action learning.Source: §1 · p. 2
  • DreamX uses learnable queries in a pretrained video diffusion model to compress 1,200 frame tokens into 4 latent-action tokens.The latent actions condition future-frame reconstruction alongside the initial frame, using the model’s generative prior to focus the representation on motion.Source: §1 · p. 2
  • DreamX achieves state-of-the-art performance among latent action models on LARYBench and supports any-to-any video and human-to-robot action transfer without paired training data.The learned latent actions can also serve as prediction targets for downstream policy learning.Source: §1 · p. 2

2 RELATED WORKS

Prior latent action models learn compact representations from visual transitions, while video generative models acquire temporal and semantic structure from scene evolution. DreamX reads actions directly from pretrained video-model representations rather than relying on an external encoder trained from scratch.

Latent action models

  • Latent action models learn representations from visual transitions, but many train from scratch on a local frame pair and can entangle action with appearance.The related-work account identifies limited temporal context and embodiment-specific representations as concerns.Source: §2 · p. 3

Emergent capabilities of video generative models

  • Video generative models learn spatiotemporal structure, including motion, geometry, and semantics, through modeling how scenes evolve over time.Their representations jointly capture which entities are present, where they are, and how they move.Source: §2 · p. 3

Video generative priors for latent action learning

  • Unlike prior approaches that use video models as pixel or rollout priors while training a separate action encoder, DreamX reads latent actions directly from the pretrained backbone.The paper states this avoids requiring a per-embodiment action encoder.Source: §2 · p. 3

3 METHOD

DreamX uses a pretrained video diffusion backbone to encode observed dynamics as latent actions and to supervise them through future-frame reconstruction. The resulting representation supports motion transfer across embodiments and compact-code prediction for robot policies.

3 METHOD

  • DreamX’s method combines pretrained-video feature extraction, future-frame reconstruction, motion transfer, and policy prediction of compact latent-action codes.For control, the policy predicts latent actions and low-level actions without generating future video.Source: §3 · p. 3

3.1 PRELIMINARIES: VIDEO DIFFUSION MODELS

  • The pretrained video diffusion model learns spatiotemporal priors through video generation, with future-video generation conditioned on an initial frame and an additional signal.The diffusion objective predicts flow velocity; the initial frame supplies scene appearance while additional conditioning steers scene evolution.Source: §3.1 · pp. 3–4

3.2 LEARNING UNIVERSAL LATENT ACTIONS FROM VIDEO DIFFUSION PRIORS

  • Learnable queries read the video backbone’s spatiotemporal representations as latent actions, while a decoder reconstructs the future from those actions and the initial frame.Query-to-video attention is frame-causal, and future-frame reconstruction encourages the actions to encode motion rather than appearance already supplied to the decoder; action chunks are decoded as an auxiliary task when labels are available.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 MOTION TRANSFER VIA LATENT ACTIONS

  • The same extracted source latent action can condition either target-scene video generation or executable actions for a target robot, without paired source–target datasets.Cross-embodiment transfer is not an explicit training objective and requires no additional training.Source: §3.3 · p. 5

3.4 POLICY LEARNING WITH COMPACT LATENT ACTIONS

  • For policy learning, a variational bottleneck compresses latent-action tokens into compact codes that the policy predicts from the current frame and instruction.A separate action head predicts low-level actions; human videos can supervise code prediction without robot action labels, while robot demonstrations also supervise actions.Source: §3.4 · pp. 5–6, Figure 3 · p. 5

4 EXPERIMENTAL RESULTS

DreamX captures motion in compact latent actions, then tests their transfer across embodiments and their use in robot policies. Results cover LARYBench, video and action transfer, policy learning, and analyses of the learned representations.

4.1 REPRESENTATION QUALITY

  • DreamX reduces LARYBench regression MSE from 0.38 to 0.17 and raises classification accuracy from 62.25% to 69.67%, outperforming the strongest baseline, DiLA.FAE-32, the compressed representation, also outperforms all baselines.Source: §4.1 · p. 6

4.2 VIDEO TRANSFER

  • DreamX transfers source motion across environments and embodiments while preserving the target scene and appearance, including transfers from humans to robots.Figure 4 includes transfers across environments for the same robot, between robot embodiments, and from humans to robots.Source: Figure 4 · p. 7, §4.2 · p. 7
  • In zero-shot human-to-robot transfer, DreamX outperforms AdaWorld and VideoWorld 2 on every metric without paired human–robot training data.The baselines show visual degradation or occasional human-hand leakage, while DreamX preserves source motion and target appearance.Source: §4.2 · pp. 7–8, Figure 5 · p. 8

4.3 ACTION TRANSFER

  • Without paired human–robot training data, robots reproduce demonstrated writing strokes, direction changes, temporal order, and corresponding bimanual movements from latent actions.The transfer is open-loop: latent actions specify motion, with proprioception supplying the robot’s initial state.Source: §4.3 · p. 8

4.4 POLICY EVALUATION

  • DreamX achieves higher success rate than Mimic-Video with 19.1× faster inference; its drawer-closing success rate averages 96.7%.Pushing completion is 80%, 100%, and 100% for the upper, lower, and both-open configurations, respectively.Source: §4.4 · pp. 8–9

4.5 ABLATION STUDY

  • Action supervision changes regression MSE from 0.24 to 0.23 and classification accuracy from 65.3% to 64.8%, suggesting video alone captures motion and action semantics.The ablation uses the same subsets of MolmoAct2, EgoVerse, and EgoDex.Source: §4.5 · p. 9

4.6 ANALYSIS

  • Perturbing latent actions disrupts manipulation while leaving the scene largely unchanged; attention maps highlight moving hands and grippers during encoding.Original latent actions reconstruct grasping and lifting, whereas zeroing them largely suppresses motion.Source: §4.6 · pp. 9–10, Figure 7 · p. 9, Figure 8 · p. 10

5 CONCLUSION

DreamX uses pretrained video diffusion priors to learn transferable latent actions and reports benchmark, transfer, and policy-learning results. The conclusion also notes deployment boundaries and responsible-use considerations.

5 CONCLUSION

  • DreamX leverages pretrained video diffusion priors to learn universal latent actions, achieving state-of-the-art performance among latent action models on LARYBench.It also supports any-to-any video transfer and human-to-robot action transfer without paired training data, and compact latent actions serve as policy prediction targets.Source: §5 · p. 10
  • The authors report using generative AI tools for manuscript editing, code editing, and research discussions, while taking responsibility for the final content.They state that AI-assisted text and code were reviewed and code changes tested for correctness.Source: AI USE STATEMENT · p. 11
  • Cross-embodiment transfer and robot policies may fail outside evaluated settings, so safety validation is needed before deployment.Human videos may contain identifiable people or surroundings; redistributed examples should follow source-dataset terms and privacy requirements.Source: ETHICS STATEMENT · p. 11

A. IMPLEMENTATION DETAILS

The appendix contents identify where implementation details, evaluation protocols, and additional quantitative results are documented.

A. IMPLEMENTATION DETAILS

  • Appendix A covers training data, DreamX pretraining, latent-action compression, and policy training and inference.These are listed as subsections A.1 through A.4.Source: A. IMPLEMENTATION DETAILS · p. 14

B. EXPERIMENTAL SETUP AND EVALUATION PROTOCOLS

  • Appendix B lists real-world robot setup, action-transfer and policy-evaluation details, and LARYBench and zero-shot human-to-robot evaluation protocols.The contents list these topics as subsections B.1 through B.5.Source: B. EXPERIMENTAL SETUP AND EVALUATION… · p. 14

C. ADDITIONAL QUANTITATIVE RESULTS

D. ADDITIONAL QUALITATIVE RESULTS AND ANALYSIS

The appendices provide supplementary transfer examples and implementation details, including training, compression, policy inference, robot setup, and action-transfer protocols.

D. ADDITIONAL QUALITATIVE RESULTS AND ANALYSIS

  • Supplementary materials include additional any-to-any video-transfer and human-to-robot action-transfer results.The appendix also identifies policy rollouts and query-to-video attention as additional qualitative material.Source: D. ADDITIONAL QUALITATIVE RESULTS AND… · p. 14

A.1 TRAINING DATA

  • DreamX pretraining uses 2.19k hours of video across six datasets, including 469 hours of human demonstrations from EgoDex and EgoVerse.Robot demonstrations come from Bridge, DROID, AgiBot, and MolmoAct-BimanualYAM.Source: §A.1 · p. 14, Figure 9 · p. 15

A.2 DREAMX PRETRAINING

  • DreamX is initialized from Cosmos-Predict2.5 2B and trained for 78,000 steps at 480 × 640 resolution, using frozen backbone weights with separate rank-16 LoRA modules.Clips contain 5–29 frames; video and action losses are equally weighted for action-labeled clips, while unlabeled clips use video loss only.Source: §A.2 · p. 14

A.3 LATENT-ACTION COMPRESSION

  • The variational bottleneck compresses each 2,048-dimensional latent-action token to a 32-dimensional code while preserving token layout.A six-layer transformer decoder reconstructs the original features; the bottleneck is later frozen for downstream deterministic codes.Source: §A.3 · pp. 14–15

A.4 POLICY TRAINING AND INFERENCE

  • The policy generator predicts compact latent-action codes conditioned on the current observation and language instruction, using about one-third of pretraining data.The Action DiT is introduced during task-specific policy learning and trained jointly with the generator.Source: §A.4 · p. 15

B.1 REAL-WORLD ROBOT SETUP

  • Action-transfer and policy experiments use a bimanual YAM robot with 14-dimensional absolute commands and a top-view Intel RealSense D405 camera.The command space includes six joint positions and one gripper command per arm.Source: §B.1 · p. 16, Figure 10 · p. 16

B.2 ACTION TRANSFER DETAILS

  • Human-video latent actions are decoded into robot commands and executed open-loop from the robot’s initial proprioceptive state; tests cover writing, directional pushing, and bimanual motion.The tasks probe trajectory and stroke order, changes in direction and temporal order, and arm-specific motion transfer.Source: §B.2 · p. 16

B.3 POLICY EVALUATION DETAILS

The appendix specifies the real-world drawer-closing policy evaluation, the LARYBench representation probe, and the zero-shot human-to-robot video-transfer protocol.

B.3 POLICY EVALUATION DETAILS

  • Three drawer configurations use the instruction “close the drawer”; when both are open, the upper drawer must close before the lower one.A fixed top-view RGB camera observes the workspace.Source: §B.3 · p. 16, Figure 11 · p. 16
  • The drawer policy is trained on 45 trajectories and evaluated over 10 trials in each of three scenarios.Subgoal success averages reaching and pushing within each configuration, then weights the three configurations equally.Source: §B.3 · p. 16

B.4 LARYBENCH EVALUATION DETAILS

  • LARYBench evaluation freezes representations and trains a common attentive probe for action regression and classification.The regression benchmark’s two-frame samples are supplemented with recovered source-video sequences for video-based models.Source: §B.4 · p. 17

B.5 ZERO-SHOT HUMAN-TO-ROBOT TRANSFER DETAILS

  • Zero-shot transfer is evaluated on 1,008 clips across 21 HumanAndRobot tasks, without paired human–robot training data.The evaluation compares DreamX with AdaWorld and VideoWorld 2 using native horizons, and reports video metrics over a common 19-frame generated horizon.Source: §B.5 · p. 17

C.1 POLICY EVALUATION

DreamX achieves higher drawer-closing subgoal success rates than Mimic-Video across the three configurations. Human-only videos also support upper-drawer policy learning at 100% subgoal success.

C.1 POLICY EVALUATION

  • 90.0%, 100.0%, and 100.0% are DreamX’s per-configuration success rates, versus 50.0% for Mimic-Video in each configuration.Each rate averages reaching and pushing completion; the reported table covers real-world drawer closing.Source: §C.1 · p. 17, Table 5 · p. 17
  • 100% subgoal success on the upper-drawer task is achieved when human videos alone supervise latent-action prediction, without low-level action labels.This human-only setting applies to the upper-only configuration; robot demonstrations remain in training for the other configurations.Source: §C.1 · p. 17, Table 6 · p. 17

C.2 ABLATION STUDY

On CALVIN, conditioning the DreamX policy on latent-action features improves success at every evaluated sequence length and increases average completed sequence length.

C.2 ABLATION STUDY

  • 2.643 versus 2.378 is the average completed sequence length with versus without latent-action conditioning on CALVIN.Conditioning also improves success rates at all five evaluated sequence lengths.Source: §C.2 · p. 18, Table 7 · p. 18

C.3 REPRESENTATION ANALYSIS

The representation analysis examines shared directional structure across robot types and how latent-action perturbations affect decoded motion and appearance. The appendix also provides further transfer examples, policy rollouts, and attention visualizations.

C.3 REPRESENTATION ANALYSIS

  • Same-direction latent-action centroids show similar organization across ten robot types in a t-SNE analysis of 2,362 clips.The analysis covers six end-effector translation directions.Source: Figure 12 · p. 18, §C.3 · p. 18
  • Stronger latent-action perturbations produce larger decoded-motion changes, while static-region appearance changes remain smaller than whole-image changes.Zeroing latent actions reduces mean per-sample motion ratio to 0.456; the perturbation study uses eight clips and two decoder seeds.Source: §C.3 · pp. 18–19, Figure 13 · p. 19, Table 8 · p. 19

D.1.1 IN-DOMAIN TRANSFER

  • Additional examples show any-to-any video transfer across embodiments and environments.These examples complement the main-paper transfer figure.Source: §D.1.1 · p. 19, Figure 14 · p. 20

D.1.2 ADDITIONAL ZERO-SHOT HUMAN-TO-ROBOT TRANSFER

  • Additional HumanAndRobot comparisons show zero-shot human-to-robot video transfer under the stated evaluation protocol.The examples complement the main-paper comparison.Source: §D.1.2 · p. 19, Figure 15 · pp. 21–23

D.2 HUMAN-TO-ROBOT ACTION TRANSFER

  • Additional action-transfer examples cover writing and single-arm manipulation.They illustrate human demonstrations transferred to robot actions.Source: §D.2 · p. 19, Figure 16 · p. 24

D.3 POLICY ROLLOUTS

  • Illustrative real-world rollouts show drawer closing in the upper-only, lower-only, and both-open configurations.The figure samples six frames at equal fractions of each rollout’s duration.Source: §D.3 · p. 19, Figure 17 · p. 25

D.4 QUERY-TO-VIDEO ATTENTION

  • Additional query-to-video attention visualizations show human and robot demonstrations, with attention overlays ordered over time.The visualizations highlight attention around the acting embodiment and objects.Source: §D.4 · p. 19, Figure 18 · p. 25

Related papers

Loading 2611.05377v1…