PaperlayerView exact revision on arXiv

Source-linked AI summary

Wan: Open and Advanced Large-Scale Video Generative Models

Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, Ziyu Liu

arXiv:2503.20314v2cs.CV

TL;DR

Wan addresses the need for capable, open video foundation models by combining scalable DiT-based generation with large-scale data, a video VAE, and automated evaluation. It reports strong cross-benchmark performance for 14B and 1.3B variants, while extending access through open release and consumer-grade inference. The report also identifies limitations in human vocal sounds, fine-grained details under large motion, and the cost of large models.

  • Problem

    Existing video generation research faces challenges in modeling complex spatio-temporal dependencies, efficiently representing high-dimensional video, and evaluating outputs in alignment with human perception.

  • Method

    Wan combines DiT and Flow Matching with full spatio-temporal attention, large-scale curated data, Wan-VAE, downstream task designs, and automated evaluation methods.

  • Results

    86.22% aggregate score is reported for Wan 14B, while Wan 1.3B scores 83.96% and remains competitive with larger and commercial models.

  • Takeaways & Limitations

    Wan provides an open, versatile family spanning multiple video-generation applications, with the 1.3B model enabling inference on consumer-grade GPUs.

  • Takeaways & Limitations

    Wan remains limited in human vocal-sound generation, fine-grained detail preservation during large motion, and the computational cost of large-scale models.

Abstract

from arXiv · show

This report presents Wan, a comprehensive and open suite of video foundation models designed to push the boundaries of video generation. Built upon the mainstream diffusion transformer paradigm, Wan achieves significant advancements in generative capabilities through a series of innovations, including our novel VAE, scalable pre-training strategies, large-scale data curation, and automated evaluation metrics. These contributions collectively enhance the model's performance and versatility. Specifically, Wan is characterized by four key features: Leading Performance: The 14B model of Wan, trained on a vast dataset comprising billions of images and videos, demonstrates the scaling laws of video generation with respect to both data and model size. It consistently outperforms the existing open-source models as well as state-of-the-art commercial solutions across multiple internal and external benchmarks, demonstrating a clear and significant performance superiority. Comprehensiveness: Wan offers two capable models, i.e., 1.3B and 14B parameters, for efficiency and effectiveness respectively. It also covers multiple downstream applications, including image-to-video, instruction-guided video editing, and personal video generation, encompassing up to eight tasks. Consumer-Grade Efficiency: The 1.3B model demonstrates exceptional resource efficiency, requiring only 8.19 GB VRAM, making it compatible with a wide range of consumer-grade GPUs. Openness: We open-source the entire series of Wan, including source code and all models, with the goal of fostering the growth of the video generation community. This openness seeks to significantly expand the creative possibilities of video production in the industry and provide academia with high-quality video foundation models. All the code and models are available at https://github.com/Wan-Video/Wan2.1.

1 INTRODUCTION

Wan is introduced as an open suite of high-performance video foundation models built on DiT and Flow Matching, trained at large scale and accompanied by an openly presented training process.

  • 1 INTRODUCTION: Wan is positioned within rapid progress in video generation and the expanding open-source community of publicly available model code and weights.The introduction places Wan alongside projects such as HunyuanVideo, Mochi, and CogVideoX.
  • 1 INTRODUCTION: Wan combines DiT, Flow Matching, cross-attention text conditioning, computational optimization, and full spatio-temporal attention for video generation.The report validates this architectural paradigm at 14 billion parameters using billions of images and videos totaling O(1) trillions of tokens.
  • 1 INTRODUCTION: The report publicly releases Wan together with its large-scale data construction, VAE, training, acceleration, and automated evaluation processes.It also provides design details and experimental results intended to support development of specialized foundational video models.

2 RELATED WORK

The related work reviews closed- and open-source video generation, framing Wan as an open model that integrates core components and targets downstream applications while competing with commercial systems.

  • 2 RELATED WORK: The review divides video generation work into closed-source models and open-source community contributions.Closed-source systems are described as resource-intensive efforts targeting high-quality, professional video generation.
  • 2 RELATED WORK: The report includes generated samples demonstrating large motion, high fidelity, realistic details, bilingual Chinese and English text, and text-to-video, image-to-video, and editing capabilities.These examples connect the model comparison with multiple video-generation functions.
  • 2 RELATED WORK: Wan reports competitive or superior performance over commercial video models across internal and external benchmarks, leading in multiple attributes.The comparison is presented in the context of intense global competition in video generation.
  • 2 RELATED WORK: Open-source diffusion video models commonly combine an autoencoder, text encoder, and diffusion-trained neural network.These modules map videos to compact latent spaces, extract text embeddings, and learn video-latent distributions, respectively.
  • 2 RELATED WORK: Wan designs or selects these critical components and explores downstream tasks including repainting, editing, controllable generation, and frame-reference generation.Prior work often uses adapter-based or ControlNet-like architectures to incorporate user-specified conditions.

3 DATA PROCESSING PIPELINE

Wan’s data pipeline curates billions of images and videos through deduplication, multidimensional quality filtering, staged processing, dense captioning, and task-specific dataset construction.

  • 3 DATA PROCESSING PIPELINE: The dataset is built around high quality, high diversity, and substantial scale, covering billions of videos and images from internal and public sources.Candidate data is curated and deduplicated before downstream filtering and processing.
  • 3 DATA PROCESSING PIPELINE: Pre-training filtering covers fundamental attributes, visual quality, and motion quality to remove unsuitable data and retain natural, diverse content.Motion assessment favors significant, smooth movement while excluding static, jittery, and shaky-camera footage.
  • 3 DATA PROCESSING PIPELINE: Data proportions for motion, quality, and category are dynamically adjusted across training stages according to data throughput.This staged provisioning is illustrated in Figure 3.
  • 3 DATA PROCESSING PIPELINE: Visual-text data combines hundreds of millions of synthesized Chinese-character images with text-containing images collected from real-world data.The two processing branches target accuracy and harmony in text rendering.
  • 3 DATA PROCESSING PIPELINE: Post-training separately optimizes image visual quality and video motion quality, selecting top-ranked samples and balancing simple and complex movements across categories.The video-selection strategy emphasizes category balance and high diversity.
  • 3 DATA PROCESSING PIPELINE: An internal caption model generates dense captions using vision-language, visual Q&A, instruction, identity, counting, OCR, camera, and fine-grained-category data.OCR processing currently covers English and Chinese text, while camera-angle and camera-motion annotations address weaknesses observed in multimodal language models.

4.1 SPATIO-TEMPORAL VARIATIONAL AUTOENCODER

Wan-VAE is a compact 3D causal VAE that compresses video spatio-temporal dimensions for diffusion training, supports efficient long-video inference, and achieves strong reconstruction quality and speed.

  • 4.1 SPATIO-TEMPORAL VARIATIONAL AUTOENCODER: Wan-VAE addresses video VAE challenges by combining 3D causal architecture, spatio-temporal compression, reduced memory use, and temporal causality.The design targets efficient integration with diffusion-based DiT models.
  • 4.1 SPATIO-TEMPORAL VARIATIONAL AUTOENCODER: Wan-VAE compresses input video dimensions by 4 × 8 × 8 while expanding the latent channel dimension to 16.The first frame receives only spatial compression to better handle image data.
  • 4.1 SPATIO-TEMPORAL VARIATIONAL AUTOENCODER: Replacing GroupNorm with RMSNorm preserves temporal causality and enables feature caching, while halving input channels in spatial upsampling reduces inference memory consumption by 33%.The feature-cache mechanism improves inference efficiency.
  • 4.1 SPATIO-TEMPORAL VARIATIONAL AUTOENCODER: 127M parameters make Wan-VAE compact, reducing encoding time and memory usage for subsequent diffusion-transformer training.The model is obtained by tuning its number of base channels.
  • 4.1 SPATIO-TEMPORAL VARIATIONAL AUTOENCODER: Wan-VAE training inflates a pretrained 2D image VAE into a 3D causal model before low-resolution, short-video training and later stages.The initialization provides a spatial-compression prior and accelerates training compared with training a video VAE from scratch.
  • 4.1 SPATIO-TEMPORAL VARIATIONAL AUTOENCODER: Feature caching processes long videos chunk by chunk while preserving context across chunk boundaries for stable inference on arbitrarily long sequences.Cached historical features are integrated into causal convolutions; the mechanism also optimizes memory utilization and preserves feature coherence.
  • 4.1 SPATIO-TEMPORAL VARIATIONAL AUTOENCODER: 2.5 times faster reconstruction than HunYuan Video is reported under the same hardware environment, alongside competitive video quality and processing efficiency.The speed advantage is attributed to the small VAE design and feature-cache mechanism, especially at higher resolutions.
  • 4.1 SPATIO-TEMPORAL VARIATIONAL AUTOENCODER: Wan-VAE reconstruction comparisons cover texture, face, text, and high-motion scenes, where the method preserves details, reduces distortion, restores text, and maintains motion sharpness.The evaluated videos contain rich details and are not used in training.

4.2 MODEL TRAINING

Wan’s text-to-video model combines a VAE, diffusion transformer, and umT5 text encoder, then uses flow matching with staged image-video training. The training strategy addresses long sequences and high-resolution video through progressive optimization and post-training.

  • Architecture: Wan’s text-to-video architecture comprises Wan-VAE, a diffusion transformer, and an umT5 text encoder.The VAE maps videos into latent space before diffusion-transformer processing, while umT5 encodes input text.
  • Architecture: The diffusion transformer patchifies latent videos with 3D convolution, models spatio-temporal context, and injects text and timestep conditioning.Cross-attention embeds text conditions, while a shared MLP predicts six modulation parameters for each transformer block.
  • Pre-training: Flow matching trains a unified denoising process across image and video domains using continuous-time velocity prediction.The model receives an interpolated latent and predicts its ground-truth velocity with an MSE objective.
  • Pre-training: Direct joint training of high-resolution images and long videos is constrained by extended sequences, reduced throughput, GPU memory use, and unstable gradient variance.Wan therefore initializes 14B training with image pre-training before joint image-video optimization.
  • Post-training: Post-training preserves the architecture and optimizer while jointly training on 480px and 720px video data.Training starts from the pre-trained checkpoint.

4.3 MODEL SCALING AND TRAINING EFFICIENCY

Wan’s scaling analysis identifies attention as the dominant long-sequence bottleneck and combines sharding, context parallelism, offloading, and resource orchestration to improve training efficiency. Its 2D context-parallel design combines Ring Attention and Ulysses while preserving data-parallel scaling.

  • Workload analysis: More than 85% of Wan’s training computation comes from the DiT model, while attention can reach 95% of end-to-end time at 1 million tokens.Attention cost grows quadratically with sequence length, whereas linear-layer cost grows linearly.
  • Workload analysis: DiT activation memory scales linearly with sequence length, but a 14B DiT can exceed 8 TB of activation memory at 1 million tokens and micro-batch size 1.Only the DiT is optimized during training; the text encoder and VAE remain frozen.
  • Parallelism strategy: With 128 GPUs, the example configuration uses Ulysses=8, Ring=2, FSDP=32, and DP=4, producing a global batch size of 8b.The corresponding context-parallel size is 16 and FSDP corresponds to a batch size of 2b.
  • Parallelism strategy: Wan uses FSDP for DiT parameter sharding and context parallelism to shard activations along the sequence dimension.The strategy targets communication overlap, larger data parallelism, and lower per-GPU activation memory.
  • Parallelism strategy: Wan’s 2D context parallelism combines outer Ring Attention with inner Ulysses to mitigate slow cross-machine communication.This design is applied to the sequence dimension of DiT activations.
  • Parallelism strategy: Wan switches distributed strategies between VAE and text-encoder data parallelism and DiT data-plus-context parallelism to avoid redundant computation.Before context parallelism, data is loop-broadcast across the context-parallel group so devices receive the same batch without duplicated upstream reads.
  • Optimization: For long sequences, activation offloading can overlap PCIe transfers with computation from 1 to 3 DiT layers.The approach exploits quadratic compute growth relative to linear memory growth.
  • Optimization: Alibaba Cloud scheduling, slow-machine detection, self-healing, and automatic task restart support stable large-scale training.Faulty nodes are isolated and repaired while training resumes automatically.

4.4 INFERENCE

Wan reduces video-generation inference cost through distributed execution, caching, quantization, and attention-kernel optimization. These techniques improve latency and memory efficiency while retaining generation quality.

  • Distributed Inference: Context Parallel reduces single-video latency across GPUs, while FSDP model sharding mitigates memory constraints for Wan 14B.FSDP is selected because long inference sequences create lower communication overhead than tensor parallelism and permit computation overlap.
  • Diffusion Cache: Diffusion caching reuses attention and classifier-free-guidance computations across sampling steps, with residual compensation preserving fine details.The cache is configured using validation data and tailored to Wan for lossless performance.
  • Diffusion Cache: 1.62×: diffusion caching improves inference performance for the Wan 14B text-to-video model.
  • Quantization: FP8 GEMM provides twice the performance of BF16 GEMM and a 1.13× speedup in the DiT module with minimal performance loss.The method uses per-tensor weight quantization and per-token activation quantization during sampling.
  • Quantization: 95% MFU: optimized 8-bit FlashAttention reaches this utilization on NVIDIA H20 GPUs and boosts inference efficiency by more than 1.27×.The optimization adjusts block size to reduce register spilling caused by Float32 accumulation across blocks.

4.5 PROMPT ALIGNMENT

Wan aligns user prompts with training-caption distributions through diverse caption augmentation and LLM-assisted rewriting. The rewriting process adds scene detail and motion attributes without changing the original meaning.

  • Caption Augmentation: Multiple captions with varied lengths and styles create diverse text-video mappings that cover a broader range of user prompts.
  • Motivation: Prompt alignment targets the mismatch between concise user inputs and longer training captions, which can adversely affect generated-video quality.
  • Prompt Rewriting: LLM rewriting adds scene details and natural motion attributes while preserving the original prompt meaning and following post-training caption structure.
  • Prompt Rewriting: Qwen2.5-Plus is selected as the rewriting model because strong instruction-following LLMs generated suitable prompts that enhanced video generation while balancing speed and performance.

4.6 BENCHMARKS

Wan-Bench is introduced as an automated, comprehensive, human-aligned evaluation framework because existing FVD and FID metrics do not adequately match human perception. It evaluates video generation across three dimensions and 14 fine-grained metrics.

  • Benchmark Design: Wan-Bench addresses the limited human alignment of existing FVD and FID evaluation methods with an automated comprehensive benchmark.
  • Benchmark Design: Three dimensions—dynamic quality, image quality, and instruction following—are decomposed into 14 fine-grained metrics.
  • Dynamic Quality: Dynamic quality measures motion magnitude, human artifacts, physical plausibility, smoothness, pixel-level stability, and identity consistency.
  • Image Quality: Image quality combines fidelity and aesthetics using MANIQA, a LAION-based aesthetic predictor, and MUSIQ.The final image-quality score averages the three evaluators.
  • Instruction Following: Scene-generation quality combines consecutive-frame CLIP similarity for temporal consistency with frame-text CLIP similarity for semantic alignment.
  • Instruction Following: Instruction-following evaluation covers stylization, object and spatial relations, camera control, and action alignment using Qwen2-VL and optical-flow analysis.

4.7 EVALUATION

Wan is evaluated against commercial and open-source models using Wan-Bench, human assessment, qualitative analysis, and the VBench leaderboard, while ablations examine key architectural components. The results report strong performance for Wan 14B and competitive performance for Wan 1.3B, alongside advantages for shared AdaLN, umT5, and the proposed VAE.

  • Benchmark Results: Wan-Bench evaluates 1,035 samples per model across dynamic quality, image quality, and instruction-following accuracy using human-preference-weighted total scores.
  • Qualitative Results: Wan generates complex dynamic scenes, physical interactions, artistic styles, cinematic visuals, and Chinese and English text from textual descriptions.
  • Human Evaluation: Wan 14B consistently excels across alignment, image quality, dynamic quality, and overall quality in human evaluation.The evaluation uses over 700 tasks annotated by more than 20 individuals.
  • Public Leaderboard: 86.22%: Wan 14B achieves a benchmark-leading aggregate VBench score, including 86.67% visual quality and 84.44% semantic consistency.
  • Public Leaderboard: 83.96%: Wan 1.3B remains competitive and surpasses HunyuanVideo, Kling 1.0, and CogVideoX1.5-5B.
  • Ablation Study: Full-shared-AdaLN-1.5B achieves the lowest training loss among equal-sized configurations, motivating shared AdaLN with greater network depth.
  • Ablation Study: umT5 provides more favorable text embeddings than the tested LLM-based encoders, while Qwen-VL second-to-last-layer features are comparable but use a larger model.
  • Ablation Study: The proposed VAE consistently achieves lower FID than the VAE-D variant at both evaluated training checkpoints.

5 EXTENDED APPLICATIONS

Wan extends video generation across image conditioning, controllable editing, personalization, streaming, image synthesis, and audio-aware generation. These applications combine specialized conditioning and data strategies with reported gains in quality, consistency, efficiency, and control.

  • Image-to-Video: Wan-I2V uses a first-frame condition image, a binary preservation mask, Wan-VAE latents, and CLIP image features to control video synthesis.The condition latent, noise latent, and rearranged mask are concatenated and passed through Wan’s DiT model.
  • Image-to-Video: Wan-I2V performs favorably across visual quality, motion quality, and matching when benchmarked against state-of-the-art techniques.Additional visualizations illustrate its ability to animate images into high-quality videos.
  • Controllable Generation and Editing: VACE extends Wan to controllable generation and editing by integrating images, videos, references, and masks through a Video Condition Unit.It supports either full Wan fine-tuning or a pluggable Context Adapter that leaves base-model weights unchanged while increasing model scale.
  • Controllable Generation and Editing: VCU inputs combine text prompts, context frames, and binary masks, which are tokenized with Wan-VAE and combined with noisy video tokens.Concept decoupling separates editable and retained pixels into reactive and inactive frame sequences before latent encoding.
  • Unified Applications: Wan’s single model achieves high video quality and temporal consistency across tasks while supporting capability combinations and expansion to new applications.Wan is jointly trained on image and video datasets, with an image dataset nearly ten times larger than its video counterpart, supporting strong image synthesis as well.
  • Streaming Video Generation: Streamer uses a sliding temporal window to reduce attention overhead and generate continuous videos of arbitrary length, while LCM integration delivers 10 - 20 × faster inference at 8 - 16 FPS.The combined system targets real-time applications while preserving the denoising-window mechanism.

6 LIMITATION AND CONCLUSION

Wan is publicly released with comprehensive architectural, training, data, evaluation, and downstream-application analyses. The report identifies strong capabilities alongside persistent challenges in large-motion fidelity, computational cost, and domain-specific expertise.

  • 6 LIMITATION AND CONCLUSION: Fine-grained detail preservation during large motion remains challenging, including for video generation more broadly.
  • 6 LIMITATION AND CONCLUSION: Inference on the 14B model requires approximately 30 minutes on a single head-end GPU without additional optimization.
  • 6 LIMITATION AND CONCLUSION: Wan is publicly released with detailed coverage of its architecture, training approach, data curation, evaluation methods, and empirical results.
  • 6 LIMITATION AND CONCLUSION: Wan investigates image-to-video generation, video editing, and personalized video generation across diverse scenarios.
  • 6 LIMITATION AND CONCLUSION: The 1.3B model achieves competitive performance against larger counterparts and enables inference on consumer-grade GPUs.

7 CONTRIBUTORS

The contributors are listed alphabetically by first name.

  • 7 CONTRIBUTORS: The authors are listed alphabetically by first name.
  • 7 CONTRIBUTORS: The contributor list includes Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, and Shiwei Zhang.
  • 7 CONTRIBUTORS: The contributor list includes Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, and Tingyu Weng.
  • 7 CONTRIBUTORS: The contributor list includes Tong Shen, Wei Lin, Wei Wang, Wenmeng Zhou, and Wente Wang.
  • 7 CONTRIBUTORS: The contributor list continues with Wenting Shen, Wenyuan Yu, Xianzhong Shi, and Xiaoming Huang.
  • 7 CONTRIBUTORS: Additional contributors are listed across the remaining alphabetical entries from Ang Wang through Ziyu Liu.
Loading 2503.20314v2…