PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Video Diffusion Models

Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, Yu-Gang Jiang

arXiv:2310.10647v2cs.CVcs.AIcs.LG

TL;DR

Existing surveys offer limited or image-centered coverage of video diffusion models, despite rapid advances in video-related research. This paper provides a comprehensive survey of video diffusion fundamentals, methods, experiments, datasets, metrics, and applications, reviewing more than 100 works across generation, editing, and understanding. It also identifies challenges and future trends, including limited video-text datasets and evaluation methods.

  • Problem

    Existing surveys provide coarse coverage of video diffusion models or emphasize image models, leaving limited up-to-date review of the video domain.

  • Method

    The paper systematically reviews video diffusion research across generation, editing, and video understanding, covering fundamentals, methodologies, experimental settings, datasets, and evaluation metrics.

  • Results

    The survey reviews over 100 works and provides a categorized overview together with experimental setups and comparative analyses across benchmark datasets.

  • Takeaways & Limitations

    Video diffusion research spans generation, editing, and understanding, while the field still needs larger video-text datasets and more comprehensive evaluation methods.

  • Takeaways & Limitations

    Text-to-video datasets remain relatively small, and current evaluation metrics do not comprehensively reflect video generation quality.

Abstract

from arXiv · show

The recent wave of AI-generated content (AIGC) has witnessed substantial success in computer vision, with the diffusion model playing a crucial role in this achievement. Due to their impressive generative capabilities, diffusion models are gradually superseding methods based on GANs and auto-regressive Transformers, demonstrating exceptional performance not only in image generation and editing, but also in the realm of video-related research. However, existing surveys mainly focus on diffusion models in the context of image generation, with few up-to-date reviews on their application in the video domain. To address this gap, this paper presents a comprehensive review of video diffusion models in the AIGC era. Specifically, we begin with a concise introduction to the fundamentals and evolution of diffusion models. Subsequently, we present an overview of research on diffusion models in the video domain, categorizing the work into three key areas: video generation, video editing, and other video understanding tasks. We conduct a thorough review of the literature in these three key areas, including further categorization and practical contributions in the field. Finally, we discuss the challenges faced by research in this domain and outline potential future developmental trends. A comprehensive list of video diffusion models studied in this survey is available at https://github.com/ChenHsing/Awesome-Video-Diffusion-Models.

1 Introduction

Diffusion models have become a dominant approach for image generation and are expanding across video and other computer vision tasks. Because existing surveys provide limited video coverage or emphasize image models, this survey comprehensively reviews video diffusion research.

  • Diffusion models are rapidly supplanting GANs and autoregressive Transformers as the predominant approach for image generation.Their reported strengths include controllability, photorealistic generation, and diversity.
  • Diffusion-based methods are also being applied to image editing, dense prediction, video synthesis, and 3D generation.
  • Existing surveys cover diffusion and multimodal learning broadly, while others focus on text-to-image or text-to-3D applications.
  • Prior surveys provide only coarse coverage of video diffusion models or place greater emphasis on image models.
  • This survey addresses the gap by systematically reviewing video generation, editing, and other video-understanding research, including methodologies, experiments, datasets, and applications.

2 Preliminaries

This section introduces diffusion models and their core formulations, then situates video diffusion research across generation, editing, datasets, and evaluation. It emphasizes the growing scope of video applications and the role of datasets and metrics in organizing comparisons.

  • 2.1 Diffusion Model Fundamentals: Diffusion models learn to reverse a process that gradually degrades training data, using forward perturbation and reverse generation chains.DDPMs transform data toward a prior distribution and learn transition kernels that reconstruct data through iterative sampling.
  • 2.1.1 Denoising Diffusion Probabilistic Models (DDPMs): DDPMs use a forward Markov chain with predefined noise levels and a learnable reverse Markov chain parameterized by neural networks.The forward transition uses β_t∈(0, 1), while the reverse process samples from p_θ(x_t−1|x_t) until the data state is reached.
  • 2.1.2 Score-Based Generative Models (SGMs): Score-based generative models estimate score functions across noise levels and generate samples by chaining those scores from high to low noise.A noise-conditional score network estimates ∇_x_t log q(x_t), with annealed Langevin dynamics providing an iterative sampling approach.
  • 2.1.3 Stochastic Differential Equations (Score SDEs): Score SDEs extend multi-scale perturbation to infinitely many noise scales and obtain data samples by reversing a stochastic differential equation.The forward SDE uses diffusion and drift functions, while the reverse-time process relies on known marginal scores to simulate samples from p_0.
  • 2.2 Related Tasks: Video diffusion research spans video generation, video editing, and video understanding, with this survey focusing on text-to-video, unconditional generation, and text-guided editing.The surveyed methodologies formulate tasks as diffusion generation problems or use controlled generation capabilities for downstream tasks.
  • 2.2 Related Tasks: Video generation datasets are categorized into caption-level and category-level collections, while evaluation combines quantitative metrics with human judgments of realism, coherence, and text alignment.Finetuned generation commonly uses UCF-101, Taichi-HD, and Time-lapse, whereas larger open-domain datasets include Panda-70M, VideoFactory, and InternVid.

3 Video Generation

The survey categorizes video generation into four groups and reviews each group before summarizing settings, evaluation metrics, and model comparisons.

  • Video generation is divided into text-to-video, generation with other conditions, unconditional generation, and video completion.
  • The survey provides detailed reviews for all four video-generation groups.
  • The section concludes with settings, evaluation metrics, and a comprehensive comparison of various models.

3.1 Video Generation with Text Condition

The survey reviews text-to-video diffusion methods across training strategies, temporal modeling, efficient training, personalization, conditioning, and multi-stage designs. These methods address text alignment, temporal consistency, video length, resolution, controllability, and computational cost.

  • Early T2V Exploration: Video Diffusion Models pioneered diffusion-based video generation by extending image diffusion U-Nets to 3D and jointly training on images and videos.
  • Early T2V Exploration: Make-A-Video learns visual-textual correlations from paired image-text data and motion from unsupervised video data, reducing reliance on paired video-text collection.
  • Multi-stage T2V methods: LAVIE uses cascaded base T2V, temporal interpolation, and video super-resolution stages, and validates joint image-video fine-tuning for high-quality outcomes.
  • Multi-stage T2V methods: Show-1 combines pixel-based and latent-based diffusion through key-frame generation, interpolation, and super-resolution stages for cost-effective high-resolution video generation.
  • Noise Prior Exploration: PYoCo uses correlated noise across frames through mixed and progressive noise models, which the authors validate as better suited to text-to-video tasks.
  • Efficient Training: GridDiff treats video as a grid image and uses a 2D U-Net, enabling direct extension of image generation and editing techniques to video.
  • Personalized Video Generation: AnimateDiff adds a motion module to a text-to-image base generator so personalized image models can generate diverse personalized videos without repeated video retraining.
  • Image-conditioned T2V methods: VideoGen uses a text-to-image reference image followed by cascaded latent diffusion and flow-based temporal upsampling to improve temporal resolution.

3.2 Video Generation with other Conditions

The survey covers video generation controlled by pose, depth, motion, audio, text, images, fMRI, and multimodal inputs. The reviewed methods introduce specialized encoders, control modules, motion representations, and conditioning pipelines for controllable and coherent video synthesis.

  • Other-condition video generation includes pose, sound, depth, motion, image, fMRI, and multimodal conditioning.
  • Pose-guided Video Generation: Follow Your Pose jointly uses pose and text control, with image-pose training followed by temporal learning from unlabeled videos.
  • Pose-guided Video Generation: Animate Anyone and MagicAnimate inject sequence pose information into image-to-video generation, enabling characters to follow predefined poses for dancing.
  • Motion-guided Video Generation: MCDiff conditions synthesis on a first frame and stroke-motion sequence, completing dense motion before autoregressively predicting subsequent frames.
  • Motion-guided Video Generation: MotionCtrl separates camera-motion and object-motion control modules that can be integrated into various video diffusion base models.
  • Sound-guided Video Generation: AADiff combines audio and text embeddings without additional training to generate audio-synchronized videos through frame editing.
  • Sound-guided Video Generation: TPoS integrates audio with variable temporal semantics and magnitude and outperforms widely used audio-to-video benchmarks in objective evaluations and user studies.
  • Other Conditional Generation: Generative Dynamics predicts long-term pixelwise motion in the Fourier domain from a single image and converts it into dense trajectories for rendering.

3.3 Unconditional Video Generation

Unconditional video generation produces videos within a domain without extra conditions. Research emphasizes video representations and diffusion architectures, including latent-space generation, depth-first synthesis, and explicit motion representations.

  • Unconditional video generation creates domain-specific videos without external conditions or prior information.
  • Latent-space Generation: PVDM maps pixels into a lower-dimensional latent space before diffusion denoising, reducing training and inference costs while maintaining satisfactory quality.
  • Depth-based Generation: GD-VDM first generates depth-map videos for scene layout, then uses them as conditioning to generate remaining video details.
  • Motion Representation: LEO represents motion with flow maps and combines a flow-based image animator with a Latent Motion Diffusion Model for human video generation.

3.4 Video Completion

Video completion covers enhancement and restoration, frame interpolation, inpainting, and prediction, with diffusion models supporting both fixed and variable-length video generation.

  • Video completion includes video enhancement and restoration, video frame interpolation, video inpainting, and video prediction.
  • Video Enhancement and Restoration: LDMVFI uses a conditional latent diffusion model with a VFI-specific autoencoder and deformable-kernel synthesis for frame interpolation.
  • Video Enhancement and Restoration: VIDM uses a pretrained latent diffusion model and masks to generate inpainted first-person videos, while AVID extends fixed-length inpainting to arbitrary lengths.
  • Video Prediction: Seer transfers text-to-image knowledge to text-guided video prediction through spatial-temporal attention and frame-sequential text decomposition.
  • Video Prediction: FDM introduces hierarchical sampling for long-video prediction and reports greater efficiency and better generative outcomes than autoregressive methods.
  • Video Prediction: Prediction methods also condition diffusion on masked frames, local-global context, or recurrent neural-network context to generate variable-length or residual future frames.

3.5 Benchmark Results

The survey compares zero-shot and fine-tuned video generation across standard datasets and metrics, finding diffusion methods advantageous over GAN and autoregressive baselines.

  • Benchmark comparisons cover zero-shot and fine-tuned video generation, reporting datasets, metrics, resolutions, parameters, training data, and dependencies.
  • Zero-shot T2V Generation: Zero-shot T2V evaluation uses MSRVTT and UCF-101 with FVD, CLIPSIM, FID, and IS measurements.
  • Zero-shot T2V Generation: Zero-shot methods using ChatGPT or other input conditions show a significant advantage, while additional data often improves performance.
  • Finetuned Video Generation: Fine-tuned generation is evaluated mainly on UCF-101, Taichi-HD, and Time-lapse using FVD, IS, and KVD, including 16-frame clips and frame resolutions.
  • Finetuned Video Generation: Diffusion-based methods exhibit a significant advantage over traditional GAN and autoregressive Transformer methods in fine-tuned video generation.
  • Finetuned Video Generation: Large-scale pretraining or class conditioning tends to further enhance fine-tuned generation performance.

4 Video Editing

Video editing research is organized around general text guidance, training-based and training-free adaptation, and mechanisms for preserving fidelity, alignment, and temporal consistency.

  • Video editing aims to preserve fidelity to the source, align outputs with control information, and maintain high-quality temporal consistency.
  • General Text-guided Video Editing: Text-guided editing uses either large-scale video-text training or extensions of pretrained text-to-image diffusion models to address frame consistency and temporal modeling.
  • Training-based Editing: Training-based methods add temporal layers, spatial-temporal units, attention strategies, or motion-aware propagation to control content, structure, and motion.
  • Training-based Editing: FlowVid combines depth maps with temporal flow instead of treating flow as a hard constraint, enabling consistent and flexible editing.
  • Training-free Editing: Training-free methods avoid heavy training costs but may produce spatio-temporal distortion and inconsistency, while T2V-based approaches can retain high costs.
  • Training-free Editing: Long-video consistency is addressed through temporal co-denoising, hierarchical cross-frame constraints, keyframe propagation, optical-flow attention, and feature-space consistency.

4.2 Other Modality-guided Video Editing

Modality-guided editing incorporates instructions, sound, coding-inspired controls, multimodal protagonist changes, and personalized appearance or motion manipulation.

  • Modality-guided editing extends beyond text to instruct-guided and sound-guided video editing.
  • Instruct-guided Video Editing: InstructVid2Vid uses ChatGPT, BLIP, and Tune-A-Video to construct video-instruction-edit triplets, with Frame Difference Loss promoting temporal consistency.
  • Sound-guided Video Editing: Soundini combines local sound and optical-flow guidance, while SDVE uses feature concatenation for temporal coherence.
  • Modality-guided Video Editing: VideoControlNet models videos as I-, P-, and B-frames, generating P-frames with motion-guided optical-flow control.
  • Multimodal Video Editing: Make-A-Protagonist combines captioning, visual and textual clue encoding, ControlNet, and mask-guided denoising for annotation-free protagonist editing.
  • Domain-specific Video Editing: CCEdit decouples structure and appearance, whereas DreamVideo separates subject learning from motion learning for more flexible personalization.
  • Domain-specific Video Editing: Domain-specific techniques include recoloring, style transfer, identity-motion-background decomposition, and 3D neural head-avatar editing.

5 Video Understanding

Video diffusion models extend beyond generation and editing into temporal localization, anomaly detection, retrieval, captioning, recognition, and soundtrack generation. The survey also situates these applications within broader challenges involving data, realism, consistency, evaluation, and benchmarking.

  • Temporal Action Detection & Segmentation: Video diffusion models support temporal action detection and segmentation by iteratively generating action segments from noise conditioned on video features.DiffAct validates this approach on GTEA, 50Salads, and Breakfast, while DiffTAD explores related temporal action detection.
  • Video Anomaly Detection: Diffusion-based anomaly detection reconstructs or predicts future motion, identifying anomalies through deviations from learned or generated patterns.MoCoDAD aggregates diverse future skeleton motions, while Diff-VAD and CMR use reconstruction capabilities for unsupervised detection.
  • Text-Video Retrieval: Diffusion models address text-video retrieval and moment retrieval by generating joint distributions or denoising noisy temporal intervals into localized video segments.DiffusionRet combines generative and contrastive objectives, while MomentDiff and DiffusionVMR map random positions to textual-query-aligned intervals.
  • Other Video Understanding Tasks: Video diffusion priors are also applied to captioning, recognition, and soundtrack generation across additional video understanding and multimodal tasks.GenRec combines a pretrained SVD backbone with classification, while LORIS generates music soundtracks synchronized with visual rhythm; RSFD targets long-tail captioning.
  • Challenges and Future Trends: The survey’s broader challenge framework highlights high-quality data, physical realism, long-video consistency, reliable evaluation, and comprehensive benchmarks.These themes are summarized qualitatively in Fig. 7.

6 Challenges and Future Trends

The survey identifies unresolved challenges in video diffusion, especially insufficient video-text data, limited evaluation coverage, and inadequate metrics. It proposes improving dataset quality and scale, constructing tailored benchmarks and metrics, and pursuing more controllable video editing.

  • Collecting Large-scale Video-Text Datasets: Text-to-video datasets remain smaller and lower quality than the image-text datasets that supported major text-to-image progress.WebVid contains 10 million instances at 360P with watermark artifacts, motivating improvements in scale, annotation accuracy, and video quality.
  • Benchmark and Evaluation Methods: Existing open-domain video-generation benchmarks and evaluation methods are limited in scope.The survey identifies benchmark construction and evaluation as continuing research needs.
  • Benchmark and Evaluation Methods: Without ground truth for generated videos, FVD and IS mainly compare generated and real distributions rather than fully measuring video quality.User AB testing and subjective scoring remain common but are labor-intensive and potentially biased.
  • More Controllable Video Editing: Future work should develop tailored evaluation benchmarks and metrics while pursuing more controllable video editing.The survey notes that current editing models achieve impressive style-transfer results but remain limited for certain tasks.

7 Conclusion

This survey reviews video diffusion models in the AIGC era, covering their foundations, datasets, metrics, and applications across generation, editing, and understanding. It synthesizes over 100 works, compares experiments across benchmarks, and identifies future research directions.

  • Scope and Foundations: The survey provides an in-depth review of video diffusion models and introduces foundational diffusion concepts, benchmark datasets, and evaluation metrics.It presents these materials as a foundation for studying the field’s latest developments.
  • Literature Review: It reviews over 100 works in video generation, editing, and understanding, categorizing them by technical perspectives and research objectives.The categorization spans the survey’s three principal application areas.
  • Experimental Analysis: The experimental section describes experimental setups and conducts comparative analysis across various benchmark datasets.The survey characterizes this comparison as fair.
  • Future Directions: The survey concludes by proposing several research directions for future video diffusion model development.These directions follow its review of current methods, benchmarks, and evaluation practices.
Loading 2310.10647v2…