PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoWorld 2: Learning Transferable Knowledge from Real-world Videos

Zhongwei Ren, Yunchao Wei, Xiao Yu, Guixun Luo, Yao Zhao, Bingyi Kang, Jiashi Feng, Xiaojie Jin

arXiv:2602.10102v1cs.CV

TL;DR

Learning transferable task knowledge from raw, unlabeled videos remains difficult because real-world environments combine complex dynamics with substantial visual variation. VideoWorld 2 addresses this with dLDM, which separates action dynamics from appearance modeling and autoregressively models the resulting latent codes. It reports improved long-horizon handicraft performance and transferable robotic manipulation knowledge across environments.

  • Problem

    Real-world videos contain complex, diverse, long-horizon interactions, while prior video-learning approaches provide limited evidence of transferable knowledge acquisition in such settings.

  • Method

    VideoWorld 2 uses dLDM to compress task-relevant visual changes into latent dynamics codes while a pretrained VDM handles appearance modeling; an autoregressive model predicts codes for task execution.

  • Results

    VideoWorld 2 achieves higher success on long-horizon Video-CraftBench tasks and substantially improves robotic action prediction on CALVIN after Open-X pretraining.

  • Takeaways & Limitations

    The results support learning transferable world knowledge directly from raw videos by separating visual appearance from core action dynamics.

Abstract

from arXiv · show

Learning transferable knowledge from unlabeled video data and applying it in new environments is a fundamental capability of intelligent agents. This work presents VideoWorld 2, which extends VideoWorld and offers the first investigation into learning transferable knowledge directly from raw real-world videos. At its core, VideoWorld 2 introduces a dynamic-enhanced Latent Dynamics Model (dLDM) that decouples action dynamics from visual appearance: a pretrained video diffusion model handles visual appearance modeling, enabling the dLDM to learn latent codes that focus on compact and meaningful task-related dynamics. These latent codes are then modeled autoregressively to learn task policies and support long-horizon reasoning. We evaluate VideoWorld 2 on challenging real-world handcraft making tasks, where prior video generation and latent-dynamics models struggle to operate reliably. Remarkably, VideoWorld 2 achieves up to 70% improvement in task success rate and produces coherent long execution videos. In robotics, we show that VideoWorld 2 can acquire effective manipulation knowledge from the Open-X dataset, which substantially improves task performance on CALVIN. This study reveals the potential of learning transferable world knowledge directly from raw videos, with all code, data, and models to be open-sourced for further research.

1 Introduction

VideoWorld 2 studies transferable knowledge learning from unlabeled real-world videos, where complex visual variation and long-horizon interactions challenge existing methods. It introduces dLDM and Video-CraftBench to learn and evaluate task-relevant dynamics across handicraft and robotic environments.

  • Motivation: Real-world videos contain dynamics, spatial relationships, and physical information that text alone cannot fully capture, motivating visual knowledge learning.The paper frames transferable knowledge from raw video as relevant to solving tasks across diverse scenarios.
  • Challenge: Real-world handicraft videos require extracting task-critical actions and planning correct sequences across deformable materials, viewpoint changes, occlusions, and diverse environments.These properties make fine-grained, long-horizon generalization a demanding evaluation setting.
  • Limitations of prior methods: VideoWorld struggles in real-world settings because complex motion and appearance cause distorted hands, incorrect object shapes, and inconsistent environments.The resulting predictions fail to produce accurate and visually coherent action sequences.
  • Approach: VideoWorld 2 decouples task-relevant dynamics from visual appearance with dLDM, using a VQ-VAE for latent actions and a pretrained VDM for high-fidelity visual reconstruction.An autoregressive model then uses the latent codes to support policy learning and long-horizon execution.
  • Results: VideoWorld 2 produces coherent long-horizon handicraft videos in unseen environments and transfers manipulation knowledge from Open-X to improve performance on CALVIN.The paper reports substantially higher success than competing methods on Video-CraftBench and robust transfer across robotic tasks.
  • Benchmark: Video-CraftBench evaluates fine-grained, long-horizon visual reasoning in real-world handicraft tasks and provides a benchmark for transferable knowledge from raw videos.The benchmark includes sequential success evaluation for long-horizon paper-folding tasks.

2 Related Work

Related work spans diffusion and autoregressive video generation, world models, and unsupervised latent-action learning. VideoWorld 2 instead targets task-irrelevant information reduction to learn transferable visual dynamics for complex, long-horizon tasks.

  • Video generation: Diffusion models emphasize photorealistic video generation, whereas autoregressive models emphasize causal sequence modeling through next-token prediction.These paradigms represent complementary priorities in video generation research.
  • Disentanglement: Prior disentanglement methods commonly separate motion and appearance for style transfer or editing, while VideoWorld 2 targets transferable dynamics for task completion.The paper distinguishes its objective from applications such as camera-motion or object-specific editing.
  • World models: World models are treated variously as controllable video synthesizers, learned dynamics simulators for planning, or abstract-space predictors that avoid pixel reconstruction.The related perspectives come from video generation, robotics, reinforcement learning, and JEPA-style research.
  • Latent actions: Unsupervised latent-action methods use cycle consistency, VQ-VAE quantization, or future-frame prediction, while paired content-similar videos remain scarce.VideoWorld 2 belongs to the broader effort to extract transferable dynamics without labeled action data.
  • VideoWorld 2: VideoWorld 2 delegates appearance modeling to a pretrained VDM so its latent space can focus on concise, transferable dynamics rather than appearance details.This design addresses VideoWorld’s appearance drift and motion errors in unseen environments.

3 Approach

VideoWorld 2 learns task-relevant dynamics from unlabeled videos by separating action dynamics from visual appearance. Its dLDM uses a pretrained video diffusion model for appearance reconstruction, while autoregressive modeling of latent codes supports transferable, long-horizon execution.

  • Generative knowledge learning: VideoWorld formulates video as demonstration trajectories whose state transitions and action policies can be learned without language supervision.The learned generator maps visual transitions into an executable policy over actions.
  • Limitations of prior models: VideoWorld’s latent dynamics model compresses future visual changes into compact codes, but real-world videos expose failures in coherence and environment generalization.Observed failures include scene drift and erroneous actions in novel handcraft environments.
  • Dynamics-enhanced latent dynamic model: VideoWorld 2’s dLDM replaces the original decoder with a pretrained VDM, allowing latent codes to focus on task-relevant dynamics rather than appearance details.The dLDM combines a causal VQ-VAE with a pretrained VDM and supplies codes through projection and causal cross-attention.
  • Dynamics-enhanced latent dynamic model: A gradient-stopped motion branch provides coarse temporal cues to the VDM, stabilizing training and supporting high-fidelity reconstruction.The cues include hand movements and object displacements, while the VDM handles visual appearance modeling.
  • Transferable dynamics: Appearance-aware latent codes show tighter intra-class alignment and reduced cross-environment variance for similar actions, indicating more transferable dynamics.This consistency is attributed to reducing the effect of appearance changes on learned codes.
  • Autoregressive transformer: VideoWorld 2 autoregressively models extracted latent-dynamics sequences and decodes predicted codes into coherent long-horizon videos in unseen environments.The transformer predicts future latent dynamics from an input frame, enabling extended action sequences beyond training observations.

4 Video-CraftBench

Video-CraftBench evaluates whether models can learn transferable knowledge from first-person tutorials of complex, long-horizon handcraft tasks. Its diverse videos and environment-shifted tests assess both action correctness and generated-video quality.

  • Benchmark scope: Video-CraftBench contains first-person tutorials for five long-horizon handcraft tasks, including paper folding and block construction.The tasks include folding paper airplanes and boats, and building block towers, horses, and people.
  • Dataset: The benchmark contains approximately 7 hours of tutorial video divided into approximately 9.5k clips, with tasks lasting 20–80 seconds.Paper-folding tasks typically last 40–80 seconds, while block-building tasks last 20–30 seconds.
  • Generalization split: Its test set includes approximately 150 separately collected videos with novel backgrounds and other appearance variations.Paper-folding tests vary background, texture, paper appearance, and viewpoint; block-building tests vary arrangement, colors, and camera angles.
  • Evaluation: Evaluation measures sequential task success and visual quality using task-step classifiers, LPIPS, and SSIM.Paper folding is decomposed into seven key steps, and a step succeeds only when all preceding steps are complete.

5 Experiments

Experiments show that VideoWorld 2 handles long-horizon real-world tasks more reliably than prior methods, while its dLDM design improves transferability, visual quality, and manipulation performance.

  • Video-CraftBench: VideoWorld 2 reaches 68.8% success on paper folding and up to 81.5% on block stacking without large-scale latent-code pretraining.It generates complete, coherent task sequences and filters background clutter, object variation, and camera noise.
  • Video-CraftBench: 72.3% final-step success on paper folding and up to 85.8% on block stacking exceed VideoWorld’s 31.9% final-step success.CoLA reaches only 40.2% on the final step, while VideoWorld 2 also transfers similar motion patterns across OpenX and Video-CraftBench.
  • CALVIN: OpenX latent pretraining substantially boosts VideoWorld 2’s success rates on CALVIN, especially for long-horizon tasks.The comparison fine-tunes on 22k CALVIN trajectories after pretraining on 1.3M OpenX examples.
  • Ablation Study: Adding the VDM prior yields an approximately 30% success-rate increase, better LPIPS, and tighter cross-environment alignment of same-action latent codes.Without VDM, codes separate strongly by environment; with it, codes align more consistently across environments.
  • Ablation Study: Using reconstructed-video motion conditioning improves LPIPS by approximately 0.9 and task success by up to approximately 20%, especially on paper folding.Stopping decoder gradients without using reconstructed video also improves success by approximately 20%, indicating decoder noise affects representations.
  • Ablation Study: N = 4 provides the best balance for dLDM query embeddings, while larger N = 8 adds noise and reduces overall success.A larger embedding length captures more information but also increases autoregressive sequence length.
  • Ablation Study: Performance improves with dLDM context length and plateaus at T = 93, whereas T = 2 lacks temporal perception for long-range paper folding.The minimal context also provides low-quality motion guidance for the VDM.
  • Ablation Study: The VDM must be adapted: a randomly initialized VDM collapses, while freezing the pretrained VDM underperforms LoRA or full fine-tuning.The authors attribute the gap to the need to capture fine-grained manipulation details specific to Video-CraftBench.

6 Further Discussion with Other Works

VideoWorld 2 differs from prior work by targeting minute-long real-world tasks and using a pretrained VDM to separate appearance modeling from task-relevant dynamics.

  • Task objectives: VideoWorld 2 targets complex, minute-long real-world tasks where appearance variation and long-horizon error accumulation dominate.This setting exposes appearance interference as a critical bottleneck.
  • Disentanglement: Prior VAE-style reconstruction objectives can force latent codes to encode task-irrelevant details, degrading long-sequence performance.Replacing the dLDM with prior tokenizers or CoLA’s training scheme significantly degrades long-sequence performance.
  • VDM role: Unlike prior VDM uses for reconstruction or rendering, VideoWorld 2 offloads appearance modeling to the VDM so latents focus on task-relevant dynamics.Its multi-frame modeling and long-horizon-oriented VDM distinguish it from CoLA.

7 Conclusion

The paper studies transferable knowledge learning from raw videos for complex, long-horizon tasks and proposes VideoWorld 2 to separate visual appearance from core actions.

  • Conclusion: VideoWorld 2 uses a pretrained VDM within a dLDM to learn generalizable, transferable policies directly from video.The experiments cover Video-CraftBench and robotic manipulation environments.
  • Conclusion: The authors identify appearance-dynamics decoupling as crucial for real-world knowledge learning and leave continued scaling for future work.The stated goal is enabling AI to learn knowledge encapsulated in the real world.

Appendix

The appendix supplies implementation details, additional Video-CraftBench analysis, and further visualizations of the method and baselines.

  • Supplementary material: Section A describes the dLDM training setup, structure, and other implementation details.
  • Supplementary material: Section B provides further analysis of Video-CraftBench.
  • Supplementary material: Section C provides additional visualizations of VideoWorld 2 and the baselines.

A Implementation Details

VideoWorld 2 combines a dynamics-enhanced latent dynamics model with autoregressive latent-token prediction. Its latent pretraining uses unlabeled videos before action fine-tuning for executable manipulation.

  • A Implementation Details: Training configurations for the dLDM and autoregressive transformer are provided in Table 4.
  • A Implementation Details: The dLDM uses a causal encoder, learnable queries, a reconstruction decoder, and a pretrained video diffusion model to separate dynamics from appearance.The encoder and queries produce latent dynamic codes; the decoder provides coarse motion cues, while the pretrained diffusion model generates high-fidelity future frames.
  • A Implementation Details: Latent pretraining trains an autoregressive transformer to predict quantized latent dynamic embeddings from task trajectories, language instructions, and the first image.The transformer vocabulary is extended with latent dynamic tokens, and the procedure does not require ground-truth actions.
  • A Implementation Details: After pretraining, an MLP action head maps transformer hidden states to executable actions using an ℓ2 loss.Fine-tuning uses ground-truth labels and only CALVIN static-camera images for cross-environment compatibility.

B Details in Video-CraftBench

Video-CraftBench evaluates long-horizon, fine-grained manipulation across varied environments. Its task classifier measures key-step completion while separately assessing visual quality.

  • B Details in Video-CraftBench: Video-CraftBench defines seven key paper-folding steps for sequential task-success evaluation.The key steps are illustrated schematically in Fig. 6.
  • B Details in Video-CraftBench: The test environments vary backgrounds, textures, paper appearance, viewpoints, block arrangements, color combinations, and camera angles relative to training.These variations assess generalization to new environments.
  • B Details in Video-CraftBench: 37.3% of tasks last 45–60 seconds, 27.1% last 60–90 seconds, and 10.9% last 20–30 seconds.Paper folding accounts for 55.3% of total duration, or 5.2 hours, while block building comprises 1.8 hours.
  • B Details in Video-CraftBench: The benchmark’s training configurations are summarized in Table 4.
  • B Details in Video-CraftBench: The classifier detects key-step completion from paper shapes and block arrangements while disregarding texture, color, and background drift.Visual quality is assessed separately with LPIPS and SSIM.
  • B Details in Video-CraftBench: The classifier is fine-tuned on approximately 25k labeled frames collected from benchmark environments and manually verified generated trajectories.

1 Function

The function implementation iterates through encoder and decoder layers, repeats the first frame for reconstruction, and combines reconstruction with diffusion-model loss.

  • 1 Function: The encoder function processes the video through its encoder layers.
  • 1 Function: The algorithm returns the sum of reconstruction MSE and the VDM loss conditioned on the video, latent code z, and reconstruction.

C Visualizations

VideoWorld 2 produces coherent long-horizon execution videos in novel environments using dLDM-based generation. Because the diffusion model processes only 93 frames at once, full sequences are extended autoregressively.

  • C Visualizations: VideoWorld 2 generates accurate, visually coherent, high-quality execution sequences in novel environments.
  • C Visualizations: The VDM processes 93 frames at a time, while complete task sequences can span thousands of frames.
  • C Visualizations: Full task videos are generated autoregressively by extending each video segment.
Loading 2602.10102v1…