Source-linked AI summary
minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models
Min Zhao, Hongzhou Zhu, Bokai Yan, Zihan Zhou, Yimin Chen, Wenqiang Sun, Kaiwen Zheng, Guande He, Xiao Yang, Chongxuan Li, Fan Bao, Jun Zhu
TL;DR
Real-time interactive video world models need causal, controllable, low-latency rollout, but existing conversion techniques remain fragmented across the required pipeline. minWM unifies camera-control fine-tuning and few-step autoregressive distillation, achieving camera-controllable generation and up to a 223.75× first-frame latency reduction over a multi-step baseline.
Problem
Existing methods for converting video foundation models into interactive world models remain fragmented, leaving a unified, reproducible, extensible full-pipeline framework missing.
Method
minWM fine-tunes bidirectional T2V/TI2V models for camera control and distills them through autoregressive diffusion training into few-step interactive generators.
Results
223.75× first-frame latency reduction over the multi-step bidirectional HY1.5 baseline accompanies preserved camera-controllable generation in the final few-step AR model.
Takeaways & Limitations
Accurate prescribed camera trajectories enable camera-controllable generation, while few-step autoregression supports substantially lower-latency interactive rollout.
Takeaways & Limitations
Under the current setup, SpatialVid-based training did not achieve reliable camera-controllable generation, possibly because perception-estimated poses introduce noise or trajectory inconsistency.
Abstract
from arXiv · showhide
Recent video diffusion foundation models have achieved remarkable progress in high-quality video generation, yet turning them into real-time interactive video world models remains challenging. Interactive world models require controllable, causal, and low-latency rollout, which in practice demands a full pipeline spanning data construction, controllable fine-tuning, autoregressive training, few-step distillation, and streaming inference. In this work, we present minWM, a full-stack open-source framework for building real-time interactive video world models. minWM provides an end-to-end pipeline that converts existing bidirectional T2V/TI2V video foundation models into camera-controllable few-step autoregressive world models. Specifically, minWM first fine-tunes a bidirectional video diffusion model with camera control, and then applies the Causal Forcing / Causal Forcing++ pipeline, including AR diffusion training, causal ODE or causal consistency distillation, and asymmetric DMD, to distill it into a few-step autoregressive generator for low-latency rollout. The framework is modular and architecture-extensible: we instantiate it on representative open backbones, including Wan2.1-T2V-1.3B and HY1.5-TI2V-8B, covering both cross-attention-based condition injection and MMDiT-style architectures. minWM also supports adapting existing video world models, such as HY-WorldPlay, to new data distributions, training recipes, and latency targets. Beyond releasing runnable scripts, checkpoints, documentation, and inference code, we provide practical ablations on camera trajectory quality, controllability training steps, and minimal batch-size requirements. We hope minWM serves as a reproducible and extensible recipe for building and adapting real-time interactive video world models. Project Page: [https://github.com/shengshu-ai/minWM](https://github.com/shengshu-ai/minWM)
1. Introduction
minWM addresses the gap between high-quality offline video generation and real-time interactive world modeling by providing a unified, reproducible, full-stack pipeline. It converts existing T2V/TI2V foundation models into camera-controllable few-step autoregressive world models across multiple open architectures.
- Motivation: High-quality offline video generators do not directly provide interactive world models, while existing conversion techniques remain scattered across separate pipelines.Building an interactive model therefore involves data construction, controllable fine-tuning, autoregressive training, few-step distillation, post-training alignment, and inference.
- Framework: minWM releases a reproducible end-to-end pipeline covering camera-conditioned data construction, controllable fine-tuning, autoregressive diffusion training, distillation, and low-latency inference.The framework converts existing T2V or TI2V video foundation models rather than releasing only a single trained checkpoint.
- Two-phase recipe: The recipe first fine-tunes a bidirectional video diffusion backbone on camera-annotated or camera-generated data, then applies Causal Forcing or Causal Forcing++ to obtain a few-step autoregressive generator.The first phase enables prescribed camera trajectories while preserving the original foundation model’s visual quality.
- Backbone coverage: The framework is instantiated on Wan2.1-T2V-1.3B with cross-attention-based condition injection and HY1.5-TI2V-8B with an MMDiT-style architecture.These examples demonstrate conversion across multiple types of video foundation models.
- Distillation pipeline: minWM supports AR diffusion training, causal ODE or causal consistency distillation initialization, asymmetric DMD post-training, and low-latency inference.These components form the stated Causal Forcing / Causal Forcing++ distillation pipeline.
2. Method
The method converts bidirectional T2V/TI2V diffusion models into camera-controllable few-step autoregressive video generators through camera-control training followed by staged AR diffusion distillation. The distillation supports causal ODE or causal consistency initialization and concludes with asymmetric DMD to align generation with the bidirectional teacher.
- Method overview: The pipeline first fine-tunes a bidirectional T2V or TI2V diffusion model for camera controllability, then distills it into a few-step autoregressive model.These are the method’s two major phases.
- Camera Control Training: PRoPE injects camera parameters into self-attention through relative projective transformations encoding camera intrinsics and poses.This preserves the backbone’s original self-attention generative structure while conditioning on camera trajectories.
- AR Diffusion Distillation: Causal Forcing or Causal Forcing++ uses three stages: AR diffusion training, causal ODE or causal consistency initialization, and asymmetric DMD.AR diffusion training uses teacher forcing to establish autoregressive generation capability, while later stages enable few-step generation and improve quality.
- AR Diffusion Distillation: Asymmetric DMD uses the bidirectional diffusion model to align the few-step autoregressive student with the bidirectional teacher’s higher-quality distribution.The student is initialized from the few-step AR model and self-rolls out a full video sequence.
- AR Diffusion Distillation: Causal consistency distillation replaces offline ODE trajectory generation to remove its data-curation time and storage overhead while remaining theoretically equivalent to causal ODE distillation.The causal CD model is trained using the AR teacher and is described as equivalent to a model obtained through causal ODE distillation.
- Camera-controllable distillation: Camera-controllable distillation propagates camera conditioning through AR training, causal ODE or CD initialization, student self-rollout, and the real and fake score models.All involved models remain camera-controllable.
3. Experiments
Experiments apply minWM to Wan2.1-T2V-1.3B and HY1.5-TI2V-8B, demonstrating large first-frame latency reductions while preserving camera controllability. Ablations identify effective ground-truth trajectory data, training duration, and batch-size requirements for controllable generation.
- Experimental setup: minWM trains Wan2.1-T2V-1.3B and HY1.5-TI2V-8B to generate 480×832 videos with 77 frames, using 4 latent-frame autoregressive chunks and 4-step distillation.HY1.5-based training generally uses batch size 32 and learning rate 1 × 10−5.
- Generation results: 223.75× and 236.64× are the first-frame latency reductions over the multi-step bidirectional HY1.5 and Wan2.1 baselines, respectively.The latency measurements use a single A800 GPU and exclude VAE-related time.
- Generation results: Few-step autoregressive models preserve camera-controllable generation and support changing the camera action.The result is presented as evidence that distillation preserves controllability.
- Training data ablation: Models trained on SpatialVid data did not reliably learn camera control, whereas reconstruction and re-rendering with prescribed trajectories enabled successful controllability learning.The authors attribute the SpatialVid difficulty hypothetically to perception-estimated camera poses; the open-source strategy also uses WorldPlay to generate videos with specified trajectories.
- Training ablations: After one to two thousand steps models remain completely uncontrollable, around five thousand steps begin to show controllability, and eight thousand steps achieve strong controllability.For Wan2.1, batch sizes smaller than 4 often fail to learn camera controllability, while batch size 8 improves it but remains somewhat unstable.
4. Conclusion and the Future Work
minWM is a full-stack open-source framework for camera-controllable video world models, supporting bidirectional T2V/TI2V fine-tuning and distillation into real-time interactive autoregressive models. Experiments show that accurate camera trajectories, sufficient training, and adequate batch size are important for reliable controllability, while future work will add controls such as pose and support more models.
- Conclusion and Future Work: minWM supports fine-tuning bidirectional T2V/TI2V models for camera-controllable generation and distilling them into real-time interactive autoregressive models, currently covering HY1.5 and Wan2.1.Future work will add control conditions beyond camera control, such as pose, and extend the framework to more models.
- Training Data: Accurate camera trajectories enable controllable generation, whereas direct training on SpatialVid did not achieve reliable controllability under the current setup.Successful training used reconstructed scenes with rendered prescribed trajectories or WorldPlay-generated images and videos with specified camera trajectories.
- Training Steps: After 1,000–2,000 steps, HY1.5-based training remained largely uncontrollable; around 5,000 steps produced unstable responses, while 8,000 steps yielded strong controllability.Controllability therefore emerges progressively during training rather than appearing immediately.
- Batch Size: Batch sizes below 4 often failed to learn camera controllability in Wan2.1, while batch size 8 improved controllability but remained unstable.The accompanying experiment identifies batch size as a critical factor in camera-control training.