PaperlayerView exact revision on arXiv

Source-linked AI summary

Vid2World: Crafting Video Diffusion Models to Interactive World Models

Siqiao Huang, Jialong Wu, Qixing Zhou, Shangchen Miao, Mingsheng Long

arXiv:2505.14357v3cs.CVcs.LG

TL;DR

Existing world models rely on labor-intensive in-domain action-labeled data yet can produce coarse, physically limited predictions, while internet-scale action-free video offers abundant and diverse world priors. Vid2World causalizes pre-trained video diffusion models and adds causal action guidance, achieving state-of-the-art world-modeling performance across domains and supporting downstream decision-making.

  • Problem

    Existing world models require meticulous in-domain action-labeled data collection and often yield coarse predictions with limited physical realism, motivating use of abundant internet-scale action-free video.

  • Method

    Vid2World transforms pre-trained video diffusion models into autoregressive, action-conditioned world models through causalization, causal training, and action guidance.

  • Results

    Vid2World achieves state-of-the-art performance in world-modeling tasks across domains and effectively supports downstream decision-making.

  • Takeaways & Limitations

    The approach provides a general pathway for repurposing passive video diffusion models into interactive world models.

  • Takeaways & Limitations

    FID scores reported for publicly released AVID code are artificially lower because the Inception model was not set to evaluation mode.

Abstract

from arXiv · show

World models, which predict future transitions from past observation and action sequences, have shown great promise for improving data efficiency in sequential decision-making. However, existing world models often require extensive domain-specific training and still produce low-fidelity, coarse predictions, limiting their usefulness in complex environments. In contrast, video diffusion models trained on large-scale internet data have demonstrated impressive capabilities in generating high-quality videos that capture diverse real-world dynamics. In this work, we present Vid2World, a general approach for leveraging and transferring pre-trained video diffusion models into interactive world models. To bridge the gap, Vid2World systematically explores video diffusion causalization, reshaping both the architecture and training objective of pre-trained models to enable autoregressive generation. Additionally, it incorporates a causal action guidance mechanism to enhance action controllability in the resulting interactive world models. Extensive experiments across multiple domains, including robot manipulation, 3D game simulation, and open-world navigation, demonstrate that our method offers a scalable and effective pathway for repurposing highly capable video diffusion models into interactive world models.

1 INTRODUCTION

Vid2World addresses the cost and fidelity limitations of conventional world models by transferring priors from internet-scale, action-free video diffusion models. It causalizes those models, adds action conditioning, and reports improvements across diverse domains.

  • Conventional world models rely on labor-intensive in-domain action-labeled data yet often produce coarse, physically limited predictions.
  • Expanding action-labeled pretraining improves efficiency and quality only partially because acquiring action-labeled data remains costly and visual fidelity remains limited.
  • Internet-scale action-free videos provide abundant, diverse data containing rich world priors for world modeling.
  • Vid2World transfers pretrained video diffusion capabilities into interactive models, avoiding training from scratch on massive video corpora.
  • Vid2World tackles causal generation and action conditioning, enabling autoregressive, action-conditioned generation across robot manipulation, 3D games, and open-world navigation.
  • The authors report significant improvements over existing transfer approaches and state-of-the-art world models across domains.

2 RELATED WORKS

Related work frames world modeling through conditional image generation, full-sequence video generation, and foundation-model adaptation. These approaches improve fidelity or temporal coherence but leave autoregressive interaction and efficient transfer unresolved.

  • Conditional image-generation world models use history observations and actions as conditions within autoregressive frameworks for games and navigation.
  • Vid2World is presented against these lines of work as a transformation from full-sequence passive video diffusion to interactive world modeling.
  • Full-sequence video-generation approaches often improve temporal coherence but preclude autoregressive rollout in interactive environments.
  • Foundation models trained on large-scale, diverse data have motivated world-model adaptation across text, image, and video modalities.

3 PRELIMINARIES

The preliminaries define pixel-space world modeling as transition prediction in a POMDP and describe diffusion models as iterative denoisers. Video diffusion extends this framework to frame sequences with independently varying noise levels.

  • World Models: A world model estimates the next-observation transition pθ(o_t+1|o_≤t, a_≤t) from observation and action histories.
  • World Models: The environment is formalized as a POMDP in which observations derive from states, actions induce state transitions, and the model operates here on pixel inputs.
  • Diffusion Models: Diffusion models learn a target distribution by transporting Gaussian noise to data through forward noising and learned reverse denoising Markov chains.
  • Diffusion Models: Noise-prediction training commonly uses a mean-square objective after reparameterizing the diffusion process.
  • Video Diffusion Models: Video diffusion represents samples as frame sequences, with each frame indexed by time and assigned a noise level.
  • Video Diffusion Models: Independent per-frame noise sampling exposes video diffusion models to diverse noise-level combinations during training.

4 METHODS

Vid2World converts bidirectional video diffusion into causal, autoregressive generation and adds frame-level action guidance. Its method combines architectural causalization, noise-aware training, and classifier-free action steering.

  • Transformation barriers: Video diffusion is unsuitable for interactive world modeling because it uses bidirectional temporal context and lacks fine-grained frame-level action conditioning.
  • Overview: Vid2World causalizes architectures and extends training objectives to convert passive video diffusion into interactive world models.
  • Video diffusion causalization: Temporal attention becomes causal through masking, while temporal convolutions require dedicated weight-transfer strategies because their symmetric kernels mix past and future features.
  • Video diffusion causalization: Shift Weight Transfer moves all kernel weights into the past, whereas Masked Weight Transfer retains only past and present weights.
  • Video diffusion causalization: Shift transfer can cause temporal misalignment, while masking discards information encoded in future-facing weights.
  • Video diffusion causalization: Extrapolative Weight Transfer uses local linear temporal feature extrapolation to preserve the original convolution’s representations more robustly.
  • Training and sampling: Causal training uses varied per-frame noise levels matching autoregressive inference, where previously generated frames are clean and the newest frame is denoised.
  • Causal action guidance: Action conditioning addresses online, frame-level control by modeling conditional and current-action-masked unconditional scores for guidance during sampling.

5 EXPERIMENTS

Vid2World is evaluated across robot manipulation, 3D game simulation, and open-world navigation, with additional tests of real-to-sim policy evaluation and method components. It achieves strong prediction quality, including large gains over baselines in CS:GO and competitive autoregressive navigation performance.

  • Experimental setup: Vid2World is evaluated across real-robot manipulation, 3D game simulation, and open-world navigation using multiple baselines and prediction settings.The experiments include non-autoregressive, autoregressive, and one-step comparisons, plus an ablation of weight transfer and action guidance.
  • Robot manipulation: Vid2World matches or outperforms prior transfer methods in robot manipulation, including superior FVD and FID in the autoregressive setting.It is comparable or better in the non-autoregressive setting and remains competitive where other baselines cannot perform autoregressive generation.
  • Real2Sim policy evaluation: Vid2World reliably tracks real-world success trends when evaluating three RT-1 drawer-closing policy checkpoints through real-to-sim rollouts.Human evaluation verifies trajectory success, while the model estimates policy performance by interacting with the world model rather than the real world.
  • 3D game simulation: 79.9% relative performance improvement in FID and 71.1% performance gain in FVD over the best baseline are reported for CS:GO.The gains are reported across all evaluation metrics with a significant margin, indicating improved visual fidelity and semantic consistency.
  • Open-world navigation: In open-world navigation, Vid2World is on par with NWM and surpasses NWM (+Ego4D) on 4 of 6 single-step metrics despite autoregressive error accumulation.Its 16-frame prediction horizon plus 4-frame history creates a 20-frame context, exceeding the 16-frame training horizon; autoregressive rollouts are superior or comparable to NWM and outperform DIAMOND.
  • Ablation study: Action guidance improves both Extrapolative and Masked Weight Transfer, while those mechanisms outperform Shift Weight Transfer and Extrapolative transfer performs slightly better.Figure 8 reports that increasing λ initially improves action alignment, but excessive guidance causes degradation from over-sharpening artifacts.

6 CONCLUSION

Vid2World transforms passive video diffusion models into interactive world models through causalization and action guidance. The section also develops Extrapolative Weight Transfer to preserve representations while enabling causal computation.

  • Vid2World introduces video diffusion causalization and causal action guidance for autoregressive, action-conditioned world-model generation.
  • Shift Weight Transfer can produce arbitrarily large approximation error even when the input sequence is perfectly smooth and linear.
  • Extrapolative Weight Transfer approximates future features using linear regression over past feature timesteps.
  • For DynamiCrafter's kernel-size-3 temporal convolutions, the construction is restricted to m = 1 and p = 2.
  • Higher-order extrapolation may improve performance but would require more complicated technical designs.

A.3 EXTRAPOLATIVE WEIGHT TRANSFER ERROR BOUND

This appendix derives error bounds for Extrapolative Weight Transfer and proves that Causal Action Guidance corresponds to sampling from a sharpened action-steered posterior. It also explains how temporal and interventional causality jointly support controllable world simulation and discusses the pretraining-to-interaction distribution gap.

  • A.3 EXTRAPOLATIVE WEIGHT TRANSFER ERROR BOUND: Under a twice-differentiable L-smooth input function, Proposition 1 bounds Extrapolative Weight Transfer's approximation error.
  • A.3 EXTRAPOLATIVE WEIGHT TRANSFER ERROR BOUND: The proof decomposes per-term extrapolation error into Taylor, intercept, and propagated slope errors before combining them.
  • A.4 CAUSAL ACTION GUIDANCE AS PROBABILITY STEERING: Causal Action Guidance is mathematically equivalent to sampling from a posterior proportional to a history-consistent prior multiplied by an action-likelihood term raised to ω.
  • Vid2World uses temporal causality as an architectural prerequisite for online rollout and action guidance to implicitly enforce interventional causality.
  • Internet-scale videos differ from interaction data in scene composition and motion granularity, but shared physical regularities are treated as transferable priors.

A.7 EXTENDED DISCUSSION ON UTILIZING VID2WORLD FOR DOWNSTREAM TASKS

The discussion identifies inference speed and downstream-task training as important boundaries for using Vid2World. It points to faster generation, more sample-efficient reinforcement learning, and hardware acceleration as future directions.

  • Vid2World has slower inference than teacher-forcing counterparts because of its large pretrained model and iterative diffusion process.
  • The authors did not explore reinforcement-learning agent training with their world models because of limited computation resources and reinforcement learning's sample inefficiency.
  • Larger world models increase computation cost, while downstream reinforcement-learning evaluations remain absent even for several industry-level systems.
  • One-step and few-step generative methods are proposed as a route toward high-fidelity future predictions with improved inference latency.
  • More sample-efficient model-based reinforcement learning is identified as important for training policy models inside world models.
  • Hardware acceleration, including KV Cache methods and better GPU locality, is suggested to improve downstream planning, control, and reinforcement learning.

B VID2WORLD IMPLEMENTATION DETAILS

Vid2World adapts a pre-trained video diffusion model for causal, autoregressive, action-conditioned world modeling through specialized training, conditioning, inference, and evaluation procedures.

  • Training: Vid2World training samples trajectory frames, independently perturbs frames with noise, predicts noise using causal histories and prior actions, and updates the model with MSE loss.
  • Inference: Autoregressive sampling denoises frames sequentially while conditioning on previously generated frames and actions, optionally applying classifier-free action guidance with scale λ.
  • Model and training setup: The implementation uses DynamiCrafter at 320 × 512 resolution, with 1.1B trainable parameters and 100k training steps on 4 × 40GB NVIDIA A100 GPUs.
  • Conditioning: Noise conditioning uses independently sampled noise levels per frame, while action conditioning uses learned embeddings for discrete actions and linear projections for continuous actions.
  • Datasets: Experiments cover robot manipulation, game simulation, and open-world navigation using RT-1, CS:GO, and RECON datasets with domain-specific action spaces.
  • Evaluation: Video prediction evaluation includes pixel-level and semantic-level similarity metrics, including PSNR, FID, FVD, SSIM, and LPIPS.

C.3.1 IMPLEMENTATION

The implementation evaluates Vid2World against transfer and world-model baselines across robot manipulation, CS:GO, and navigation, using autoregressive policy simulation where applicable.

  • RT-1: RT-1 uses 82,851 training episodes and 4,361 holdout episodes, with continuous actions injected through a linear layer.
  • RT-1: RT-1 training uses 16-frame video-action segments, up to 100k gradient steps, and 4 × A100 GPUs, taking 6.4 days under the reported setup.
  • Baselines: Baselines share the Dynamicrafter base model but are transferred without causal architecture or homogeneous-noise changes, so they remain non-causal during training and sampling.
  • Real2Sim policy evaluation: Real2Sim evaluates policies by feeding their actions into the world model, iteratively generating imagined frames and estimating success with a trajectory verifier.
  • CS:GO: CS:GO evaluation uses the same 0.5M-frame holdout as DIAMOND, while discrete actions are injected through learned embeddings and 16-frame segments.
  • CS:GO baselines: DIAMOND provides fast and high-quality configurations that trade generated-image fidelity against inference speed.
  • Open-world navigation: RECON uses matched train-evaluation splits and linear action projection, while NWM relies on a 1B-parameter architecture trained with substantially greater computational resources.

D.1 GENERATION RESULTS OF RT-1

On RT-1, Vid2World generates high-fidelity, physically plausible predictions that represent environment dynamics, while still exhibiting fine-grained control and scene-regression failures.

  • Generation quality: Vid2World accurately represents RT-1 environment dynamics with physically plausible, high-fidelity frame sequences.
  • Failure cases: The model successfully predicts the robot arm’s moving directions but fails to capture the gripper’s control over the green bag.
  • Failure cases: In another failure mode, robot movement is mostly correct, but the grasped object changes to a more frequently seen object.
  • Visualization: Figure 10 compares ground-truth and generated videos when only the first frame is provided as context.

D.2 GENERATION RESULTS OF CS:GO

In CS:GO, Vid2World reduces rollout degradation, aligns predictions with actions, and produces divergent counterfactual sequences, but both compared models still have failure cases.

  • Error accumulation: Vid2World maintains sharp, physically accurate predictions during rapid viewpoint changes, while DIAMOND’s frames progressively blur from accumulated rollout errors.
  • Action alignment: Vid2World reflects the aim-down-sights action in predicted videos, whereas DIAMOND fails to manifest that action.
  • Failure cases: In the reported failure cases, neither Vid2World nor DIAMOND matches ground truth, with environment respawn randomness adding difficulty.
  • Action influence: Different action sequences produce completely different Vid2World frame sequences despite identical initial observations, demonstrating action-dependent generation.

D.3 GENERATION RESULTS FOR OPEN-WORLD NAVIGATION

Vid2World generates autoregressive, action-conditioned video across open-world navigation, policy evaluation, and 3D game simulation settings. The results include long-horizon extrapolation, out-of-distribution generalization, diverse stochastic predictions, and action-responsive trajectories.

  • Open-world and policy evaluation: Vid2World provides generation results for open-world navigation and Real2Sim policy evaluation.The corresponding figures compare generated videos in the open-world navigation task and Real2Sim Policy Evaluation experiments.
  • Action guidance scale: A guidance scale λ produces improving then degrading generation quality across PSNR, SSIM, LPIPS, and DreamSim.Insufficient guidance weakens action adherence, whereas excessive guidance can cause over-sharpened distributions and visual artifacts.
  • Long-horizon rollout: 100-frame CS:GO rollouts from 9 history frames demonstrate autoregressive temporal extrapolation beyond the training horizon.The rollout is conditioned on random action sequences and extends over 6x the training horizon.
  • Zero-shot generalization: Trained exclusively on CS:GO, Vid2World generates 50-frame Valorant rollouts with rudimentary temporal consistency and action responsiveness.The observed cross-domain behavior is attributed to visual priors preserved from the pretrained video diffusion backbone.
  • Stochasticity and action semantics: The model produces diverse plausible weapon outcomes under an unseen switch-weapon action and directionally coherent robot motions under distinct action sequences.The qualitative results cover stochastic future prediction in CS:GO and action-semantic following in the RT-1 environment.

D.9 QUANTITATIVE RESULTS ON INTERACTIVE METRICS

Vid2World’s interactive-metric evaluation compares generation under ground-truth and random actions using normalized delta metrics. The results show stronger action-following scores than baselines, while exposing a metric-hacking concern and separately supporting transfer of pretrained representations.

  • Metric definition and setup: Normalized delta metrics compare generation quality under ground-truth actions with quality under random actions.The metric measures the normalized difference between the two action conditions, with direction depending on whether higher or lower values are better for metric M.
  • Interactive metrics: Vid2World outperforms baseline methods in ∆N-FVD and ∆N-FID, demonstrating stronger action-semantic following.The evaluation uses autoregressive prediction from four history frames over 500 CS:GO validation trajectories.
  • Metric limitation: Random-action degradation can inflate delta-normalized metrics, creating a metric-hacking concern that separates action following from generation quality.The concern arises because a model may obtain high scores by producing low-quality videos under random action distributions.
  • Role of pretraining: Training the identical architecture from scratch causes a significant performance drop across all metrics compared with Vid2World models using pretrained video diffusion weights.This ablation supports transfer of generation priors from large-scale video pretraining.
  • Representation preservation: All three causal weight-transfer methods achieve cosine similarity above 0.7 with the original video diffusion model’s representations.The extrapolative variant achieves the highest similarity, providing evidence that representations remain close after transformation.
  • Computational scope: The current system prioritizes visual fidelity and physical realism over inference latency, leaving computational efficiency as an open boundary.The paper identifies fewer sampling steps, shorter history, software optimization, and rolling KV-cache as possible speedup routes.
Loading 2505.14357v3…