A Survey on Video Diffusion Models

1 Introduction

• Diffusion models [83] [188] have become predominant in image generation and are expanding across video synthesis and other computer-vision tasks.The paper attributes their success to strong controllability, photorealistic generation, and diversity.Source: §1 · pp. 1–2
• Since 2022, video diffusion publications have increased significantly across video generation [11] [244], editing [48] [56] [136], and understanding [25].Source: §1 · p. 2
• Existing surveys offer coarse coverage of video diffusion or emphasize image models, motivating this survey’s comprehensive review of video methods, settings, benchmarks, and applications.The authors state that, to their knowledge, this is the first survey focused specifically on video diffusion and that it compares video-generation benchmarks and settings.Source: §1 · p. 2
• The survey introduces background and preliminaries, reviews generation, editing, and understanding methods, and concludes with research challenges and future directions.The authors caution that rapid advances mean the survey may not cover all the latest work and invite new findings for future revisions.Source: §1 · p. 2

2.1 Diffusion Model

• Diffusion models [82] learn to reverse data degradation and are chiefly formulated as DDPMs [82], SGMs, or Score SDEs.The survey describes them as a leading family of generative models that has displaced GANs across several image-generation and editing tasks [8] [4] [34].Source: §2.1 · p. 3

2.1.1 Denoising Diffusion Probabilistic Models (DDPMs).

• DDPMs use a forward Markov chain to perturb data into noise and a learned reverse chain to recover data, generating samples by iterative reverse-chain sampling.The forward process has transitions controlled by a preset noise hyperparameter; the reverse chain starts from a Gaussian prior and learns transition means and variances with neural networks.Source: §2.1 · p. 3, §2.1.1 · pp. 3–4

2.1.2 Score-Based Generative Models (SGMs).

• SGMs perturb data at multiple noise levels and train one conditional score network to estimate scores, then sample by chaining score functions from high to low noise.Training and sampling are decoupled, and annealed Langevin dynamics is one stated sampling approach.Source: §2.1.2 · p. 4

2.1.3 Stochastic Differential Equations (Score SDEs).

• Score SDEs extend multi-scale perturbation to infinitely many noise scales, modeling diffusion with an SDE and sampling through its reverse-time SDE.The reverse process uses marginal-distribution scores to simulate samples from the data distribution.Source: §2.1.3 · p. 4

2.2 Related Tasks

• Video diffusion applications span video generation, video editing, and other video-understanding tasks, with this survey focusing especially on text-to-video, unconditional generation, and text-guided editing.Text-to-video converts descriptions into coherent visual frames, unconditional generation learns coherent videos without external guidance [79], and text-guided editing uses natural-language instructions to modify video content [48] [205].Source: §2.2 · pp. 4–5

2.3.1 Data.

• Video-generation datasets include caption-level and category-level collections, evolving from small, domain-specific data toward larger and higher-resolution resources.WebVid-10M [5] boosted open-domain text-to-video but has 360P resolution and watermarks; Panda-70M, VideoFactory [244], and InternVid [250] offer larger-scale, 720P data. Category-level benchmarks such as UCF-101 [214] are often small or narrow-domain, while newer purpose-built datasets target high resolution [11] or long duration [13] [282].Source: §2.3.1 · pp. 5–6

2.3.2 Evaluation Metrics.

• Video-generation evaluation combines costly human judgments of realism, coherence, and text alignment with quantitative image-level and video-level metrics.FID measures frame-distribution quality; PSNR and SSIM assess pixel-level reconstruction, while CLIPSIM measures image-text relevance. FVD and KVD use I3D video features, Video IS uses C3D features, and Frame Consistency CLIP Score averages pairwise frame-embedding similarity.Source: §2.3.2 · pp. 6–7

3 Video Generation

• The review groups video generation into text-to-video, generation with other conditions, unconditional generation, and video completion.It also summarizes settings and evaluation metrics and compares the models reviewed.Source: §3 · p. 7

3.1 Video Generation with Text Condition

• The video-generation taxonomy covers general T2V generation, domain-specific generation, conditional control generation, and video completion.Source: Figure 3 · p. 8

3.1.1 Training-based T2V Diffusion Methods.

• VDM pioneered diffusion T2V with a 3D U-Net jointly trained on images and videos, while Make-A-Video learns text-image alignment and motion from separate data.VDM also supports unconditional generation and video prediction; Make-A-Video uses super-resolution and interpolation models to increase video definition and frame rate.Source: §3.1.1 · p. 8
• Training-based methods model temporal coherence through shared and residual noise in VideoFusion and correlated frame noise in PYoCo [63], whose mixed and progressive priors suit T2V.VideoFusion’s noise decomposition aims to maintain frame-motion consistency, while PYoCo fine-tunes eDiff-I [6] for video generation.Source: §3.1.1 · pp. 9–10
• Other training-based variants target personalized and image-conditioned generation (AnimateDiff [75], VideoGen [132]), complex-action planning (Dysen-VDM [52]), and Transformer-based generation such as GenTron [23].AnimateDiff adds a motion module to a T2I base model; Dysen-VDM turns text into chronologically ordered dynamic scene graphs, while GenTron adapts diffusion Transformers to text conditioning.Source: §3.1.1 · pp. 10–11

3.1.2 Training-free T2V Diffusion Methods.

• Training-free approaches use pretrained T2I models, LLM-generated frame descriptions, or dynamic scene layouts to guide video generation without training [134].Text2Video-Zero links frames to the first frame through cross-attention; LVD [134] adjusts attention maps using LLM-generated layouts.Source: §3.1.2 · p. 12

3.2.1 Pose-guided Video Generation.

• Follow Your Pose [156] learns pose control from image-pose pairs and temporal modeling from pose-free videos, while Animate Anyone and MagicAnimate use reference characters and pose encoders for zero-shot dancing.The latter methods demonstrate zero-shot capabilities with only a small training set.Source: §3.2.1 · p. 13

3.2.2 Motion-guided Video Generation.

• MCDiff [27] predicts dense motion from stroke controls to generate frames autoregressively, DragNUWA [281] enables open-domain trajectory control, and MotionCtrl [256] separates camera- and object-motion controls.DragNUWA combines text, image, and trajectory inputs; MotionCtrl’s trained modules can integrate with different video diffusion base models.Source: §3.2.2 · p. 13

3.2.3 Sound-guided Video Generation.

• TPoS incorporates audio with variable temporal semantics and magnitude, and outperforms widely used audio-to-video benchmarks in objective evaluations and user studies.AADiff also generates audio-synchronized video without additional training by selecting text tokens using audio-text similarity; EMO [221] injects audio features through audio attention for expressive talking animations without 3D facial information.Source: §3.2.3 · pp. 13–14

3.2.4 Image-guided Video Generation.

• LaMD [91] separates video motion with an autoencoder before training a motion generator, while LFDM and Generative Dynamics [133] use image-conditioned motion representations to produce coherent or looping videos.LFDM conditions diffusion on image, flow, and text; Generative Dynamics predicts per-pixel motion in the Fourier domain from a single image.Source: §3.2.4 · p. 14

3.2.5 Brain-guided Video Generation.

• MinD-Video [31] conditions generation on fMRI data aligned with images and text, using temporal attention to reconstruct videos with precise semantics, motion, and scene dynamics, reportedly surpassing ground-truth performance.A trained MRI encoder replaces the CLIP text encoder as the conditioning input.Source: §3.2.5 · p. 14

3.2.6 Depth-guided Video Generation.

• Make-Your-Video [264] uses MiDaS-derived depth conditioning and causal attention for longer videos, with reported quantitative and qualitative advantages over state-of-the-art methods.Animate-A-Story [78] retrieves videos matching a text prompt, derives depth-map motion guidance, and uses it to synthesize personalized videos.Source: §3.2.6 · pp. 14–15

3.2.7 Multi-modal guided Video Generation.

• VideoComposer [246] flexibly combines spatial and temporal conditions, MM-Diffusion links audio and video generation, and VITRON reports good results for image-to-video and text-to-video.VideoComposer supports conditions including sketches, masks, depth, and motion vectors; MM-Diffusion also supports video-to-audio translation.Source: §3.2.7 · p. 15

U-Net based Generation

• VIDM separates content and motion streams, merging them for consistent video generation and using PosGN with IMC to improve long-video continuity.Source: §3.3 · p. 15
• PVDM diffuses in a learned lower-dimensional latent space, reducing training and inference costs while maintaining satisfactory video quality.Source: §3.3 · p. 15
• GD-VDM [122] generates depth-map videos to prioritize scene layout, then conditions detail generation on those maps, targeting complex driving scenes.Source: §3.3 · p. 16
• LEO [253] separates motion from appearance using flow maps and combines a flow-based animator with latent motion diffusion for human video synthesis.The approach also extends to infinite-length human video synthesis and content-preserving video editing.Source: §3.3 · p. 16

Transformer-based Generation

• VDT [152] and Latte [155] extend diffusion transformers to video, exploring spatio-temporal attention variants, conditional injection modules, and scaling capabilities.Source: §3.3 · p. 16

Mamba-based Generation

• Matten [60] combines Mamba and attention for spatio-temporal modeling, while DiM [163] tests pure Mamba and reports scalability, high-quality video generation, and lower computational cost.Source: §3.3 · p. 16

3.4 Video Completion

• CaDM reduces streaming bitrate by lowering frame resolution and color bit-depth, while conditioning diffusion-based decoder restoration on those encoder settings [308].The survey describes its restoration capability as heightened relative to prevailing methodologies.Source: §3.4.1 · p. 16

3.4.1 Video Enhancement and Restoration.

• LDMVFI applies conditional latent diffusion to frame interpolation, while VIDM uses LDM for masked first-person video inpainting and AVID extends inpainting to arbitrary-length videos [40] [21].LDMVFI uses a VFI-specific autoencoder with efficient self-attention and deformable-kernel synthesis; AVID uses Temporal MultiDiffusion with middle-frame attention guidance.Source: §3.4.1 · p. 16

3.4.2 Video Prediction.

• Seer transfers text-to-image priors to text-guided video prediction through spatial-temporal attention, an autoregressive framework, and a Frame Sequential Text Decomposer [73].The passage reports performance enhancements on benchmarks [46] [68].Source: §3.4.2 · pp. 16–17
• FDM uses hierarchy sampling for long-video prediction and reports greater efficiency and better generation than autoregressive methods, while introducing the CARLA dataset [77] [42].The hierarchy sampling scheme is designed for long-video prediction tasks.Source: §3.4.2 · p. 17
• MCVD masks past or future frames for prediction and generates variable-length videos block-wise, while RVD adds a diffusion-predicted residual to a deterministic next-frame prediction [278].The RVD authors report residual prediction is more effective than directly predicting future frames.Source: §3.4.2 · p. 17
• LGC-VD uses two-stage training and prediction errors as data augmentation to address errors and reinforce stability in long-video prediction.It is designed to cover diverse perceptual conditions.Source: §3.4.2 · p. 17
• RaMViD extends image diffusion with 3D convolutions and mask-conditioned training for completion tasks, while AID transfers general video diffusion to domain-specific prediction [267].RaMViD covers prediction, infilling, and upsampling [17] [45] [214]; AID uses DQFormer and a lightweight adapter to transfer SVD to robotics and first-person video generation [10].Source: §3.4.2 · p. 17

3.5 Benchmark Results.

• The section compares video-generation methods under zero-shot and finetuned settings, introducing commonly used datasets and metrics before presenting performance comparisons.Source: §3.5 · p. 17

3.5.1 Zero-shot T2V Generation.

• Zero-shot T2V evaluations use MSRVTT and UCF-101, with FVD, FID, and CLIPSIM on MSRVTT and IS, FVD, and FID on UCF-101.MSRVTT test-set descriptions serve as prompts; UCF-101 videos are generated from action categories or manually set prompts.Source: §3.5.1 · pp. 17–18
• In zero-shot T2V, methods using ChatGPT [52] or other input conditions [78] [264] show a significant advantage, while additional data [244] [250] often improves performance.Table 3 compares methods on MSR-VTT and UCF-101 and includes parameter count, training data, extra dependencies, and resolution.Source: §3.5.1 · p. 18, Table 3 · p. 18

3.5.2 Finetuned Video Generation.

• Finetuned generation benchmarks cover UCF-101, Taichi-HD, and Time-lapse, using IS and FVD on UCF-101 and FVD and KVD on the other datasets.The datasets represent human sports, Tai Chi, and sky time-lapse footage, respectively.Source: §3.5.2 · p. 18, Table 4 · p. 19
• Diffusion-based methods significantly outperform GANs and autoregressive Transformers [275]; large-scale pretraining or class conditioning further enhances finetuned results.Table 4 compares results on UCF-101, Taichi-HD, and Time-lapse, reporting FVD, IS, and KVD for 16-frame clips.Source: §3.5.2 · p. 19, Table 4 · p. 19

4 Video Editing

• Video editing is challenging because frame-by-frame image diffusion lacks cross-frame semantic consistency, and outputs must preserve fidelity, align with controls, and remain temporally consistent and high quality.The survey organizes video editing into text-guided, modality-guided, and domain-specific categories.Source: §4 · p. 20

4.1 Text-guided Video Editing

• Text-guided editing either trains a text-to-video model on large text-video datasets or adapts pretrained text-to-image models, with the latter favored when data and training costs are prohibitive.Inflated image models add temporal modules but can still produce flicker or fail to follow prompt semantics.Source: §4.1 · p. 20

4.1.1 Training-based Methods.

• Training-based methods use large video-text datasets; GEN-1 [48] jointly trains temporal layers on images and videos, while Dreamix [165] uses low-resolution initialization and mixed temporal-attention fine-tuning.TCVE [252] connects temporal and pretrained image U-Nets with a cohesive spatial-temporal unit to capture input-video coherence.Source: §4.1.1 · p. 20
• Control-A-Video [29] enables autoregressive videos of any length, while MagicEdit [136], MagicProp [274], and FlowVid [135] separate content, appearance, or motion and use spatial-temporal conditions for coherent editing.MagicProp propagates from an edited reference frame; FlowVid combines depth maps with temporal flow rather than treating imperfect flow estimates as hard constraints.Source: §4.1.1 · p. 20

4.1.2 Training-free Methods.

• Training-free methods avoid heavy training but can incur distortion or high inference costs; TokenFlow [64] propagates jointly edited keyframe features, while VidEdit [38] enables zero-shot atlas editing with object-level control.Rerender-A-Video uses optical-flow constraints anchored by the first rendered frame, and FateZero [183] stores inversion attention maps and adds spatial-temporal blocks.Source: §4.1.2 · p. 21
• Vid2Vid-Zero [243], Pix2Video [19], and InFusion combine inversion, anchor-frame propagation, or attention-feature and mask strategies to preserve prompt alignment, fidelity, and temporal consistency.Their mechanisms respectively include spatial regularization and cross-frame modeling, self-attention feature injection, and attention merging guided by cross-attention masks.Source: §4.1.2 · p. 21
• ControlVideo1 [299], EVE [30], MeDM [35], Gen-L-Video [235], and FLATTEN [36] use cross-frame interaction, depth or flow guidance, correspondence alignment, segment co-denoising, or flow-guided attention.ControlVideo1 [299] and Gen-L-Video [235] address long videos through keyframe-conditioned clips or overlapping segments, respectively.Source: §4.1.2 · pp. 21–22

4.1.3 One-shot-tuned Methods.

• One-shot-tuned methods fine-tune a pretrained image model on a specific video for flexible editing at extra training cost; SinFusion, SAVE [106], Edit-A-Video [205], Tune-A-Video, and Video-P2P [146] adapt motion, content, or attention.Edit-A-Video [205] uses sparse-causal blending to approximate edited regions, while Tune-A-Video updates attention projections and uses input-video structural guidance.Source: §4.1.3 · p. 22

4.2 Other Modality-guided Video Editing

• The survey frames instruction and sound as examples of editing guidance beyond the text-guided methods discussed earlier.Source: §4.2 · p. 22

4.2.1 Instruct-guided Video Editing.

• InstructVid2Vid [184], CSD, VIDiff [268], and Fairy [259] address instruction-guided editing through dataset construction, consistency methods, task unification, and anchor-frame propagation.InstructVid2Vid uses ChatGPT, BLIP, and Tune-A-Video to create video-instruction-edit triplets and introduces Frame Difference Loss for temporal consistency; CSD combines SVGD with Instruct-Pix2Pix [14] for coherent multi-image editing. VIDiff unifies video editing, recoloring, segmentation, and low-level tasks, while Fairy propagates anchor-frame features to similar frames for parallel processing, temporal consistency, and reduced memory use.Source: §4.2.1 · p. 23

4.2.2 Sound-guided Video Editing.

• Soundini [125] aligns sound and image latents with local sound and optical-flow guidance, while SDVE [9] uses feature concatenation and speech-conditioned spectral embeddings for temporal coherence.Source: §4.2.2 · p. 23

4.2.3 Motion-guided Video Editing.

• VideoControlNet [92] uses I-, P-, and B-frame roles to generate key frames with diffusion and interpolate others, while MotionEditor [224] enforces temporal motion correspondence.VideoControlNet uses optical-flow-guided P-frame generation and interpolates B-frames from reference I/P-frames and motion information, avoiding diffusion for those frames; MotionEditor combines reconstruction and editing branches for high-fidelity, temporally consistent editing.Source: §4.2.3 · p. 23

4.2.4 Multi-Modal Video Editing.

• Make-A-Protagonist [304], CCEdit [56], and DreamVideo [257] support protagonist, appearance, and personalized video editing through multimodal conditioning or disentangled structure, appearance, subject, and motion controls.Make-A-Protagonist uses captioning, visual and textual clues, ControlNet, and mask-guided denoising for annotation-free editing. CCEdit preserves structure with ControlNet while text prompts, personalized weights, and center frames guide appearance; DreamVideo separates subject and motion learning with lightweight adapters.Source: §4.2.4 · p. 23

4.3 Domain-specific Video Editing

• ColorDiffuser [144], Style-A-Video [95], and Diffutoon [43] target recoloring and stylization with spatiotemporal color propagation, zero-shot style control, and toon-shading stages.ColorDiffuser replaces optical flow with Color Propagation Attention and uses alternated sampling; Style-A-Video combines text, frames, and attention maps for style, content, and detail without per-video training or fine-tuning. Diffutoon separates toon shading into stylization, consistency enhancement, structure guidance, and colorization, using Stable Diffusion [193] and ControlNet [297].Source: §4.3.1 · p. 24

4.3.2 Human Video Editing.

• Human-video methods manipulate identity, emotion, texture, or motion in talking-head and avatar editing through disentangled features and multi-condition pipelines.Diffusion Video Autoencoders separate time-invariant identity from per-frame motion and background features for temporally consistent editing and efficient computation. Instruct-Video2Avatar [131] combines image editing [14], EbSynth, and INSTA for instruction-edited 3D neural head avatars; TGDM [270] uses zero-shot CLIP guidance for emotion control and multi-conditional diffusion for texture and identity transfer.Source: §4.3.2 · p. 24

Temporal Action Detection & Segmentation

• DiffTAD diffuses action proposals and reaches state-of-the-art results on ActivityNet and THUMOS, while DiffAct iteratively generates action segments from noise conditioned on video features.DiffAct is evaluated on GTEA, 50Salads, and Breakfast.Source: §5 · p. 24

Video Anomaly Detection

• Diff-VAD and CMR detect anomalies from high reconstruction error, while MoCoDAD [57] flags skeleton-motion anomalies when generated future actions diverge from observed trends.Diff-VAD and CMR significantly improve performance over prior research on two large-scale benchmarks.Source: §5 · pp. 24–25

Text-Video Retrieval

• DiffusionRet generates a joint candidate-query distribution using generative and contrastive losses, while MomentDiff and DiffusionVMR [302] denoise noisy time intervals to localize text-described moments.DiffusionRet shows strong generalization in open-domain scenarios.Source: §5 · p. 25

Video Captioning and Video Object Segmentation

• RSFD improves captioning of low-frequency tokens through refined semantic enhancement, while Pix2Seq-D [25] models video panoptic masks and learns tracking and object segmentation from prior-frame predictions.Pix2Seq-D [25] formulates panoptic segmentation as discrete data generation and models masks using a diffusion model based on analog bits [26].Source: §5 · p. 25

Video Pose Estimation, Audio-Video Separation, and Action Recognition

• DiffPose [55] refines pose keypoints through cross-frame feature aggregation and multi-scale interaction; DAVIS [93] generates separated audio magnitudes conditioned on an audio mixture and visual content.DDA [103] uses diffusion-based augmentation to generate diverse, natural skeleton-action sequences that benefit existing action-recognition models.Source: §5 · p. 25

Video Recognition

• GenRec [258] adds a classification head to pretrained SVD [10], supporting generation and recognition whose tasks can complement each other and achieve strong performance.The method applies spatiotemporal priors from a video diffusion model to recognition.Source: §5 · p. 25

Video SoundTracker and Video Procedure Planning

• LORIS generates long-term music waveforms synchronized with rhythmic visual cues, while PDPP represents intermediate instructional-video actions as a diffusion distribution sampled under initial- and final-observation guidance.LORIS uses context-aware temporal conditioning and extends to sports scenarios; PDPP uses a diffusion-based U-Net for conditional guidance.Source: §5 · p. 25

Collecting Large-scale Video-Text Datasets

• High-quality open-source video-text datasets are needed because current T2V datasets are limited in scale and quality.WebVid [5] contains 10 million 360P instances with watermark artifacts; although scaling has benefited SVD [10] and Sora, their datasets are private.Source: §6 · p. 26

Physical Realism and Long Video Generation

• Video generation still struggles with physical realism and stable long-video synthesis.Sora can misrepresent complex-scene physics; most models [11] produce videos under 10 seconds, while autoregressive [178] and coarse-to-fine [282] approaches have quality, complexity, or speed drawbacks.Source: §6 · p. 26

Efficient Training and Inference

• Reducing the training and inference costs of T2V models remains an important research direction.Some tasks require hundreds of GPUs [11] [71]; proposed directions include video-representation compression [238], spatiotemporal modeling [60], and acceleration [247] [296].Source: §6 · p. 26

Benchmark and Evaluation Methods

• Current video-generation benchmarks and metrics do not comprehensively capture quality, motivating tailored evaluation methods.FVD and IS compare generated and real video distributions without ground truth, while user AB testing and subjective scoring are labor-intensive and potentially biased.Source: §6 · p. 27

More Controllable Video Editing

• Video editing needs stronger control and generality, particularly for object replacement, multi-object, motion, and long-video edits.Existing methods can show temporal inconsistencies in object replacement and often depend on detailed text descriptions [184] [268]; future models should support greater multimodal capability.Source: §6 · p. 27

• The survey reviews diffusion fundamentals, popular benchmark datasets, and commonly used evaluation metrics as foundations for the field.Source: §7 · p. 27
• It categorizes more than 100 works on video generation, editing, and understanding by their technical perspectives and research objectives.Source: §7 · p. 27
• The survey describes experimental setups and conducts fair comparative analyses across multiple benchmark datasets.Source: §7 · p. 27
• It proposes research directions for the future development of video diffusion models.Source: §7 · p. 27