Source-linked AI summary
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, Robin Rombach
TL;DR
Video-generation research lacks a unified strategy for training stages and curating data, even though data distribution strongly affects generative models. Stable Video Diffusion studies three training stages and systematic video curation, producing strong text-to-video, image-to-video, and multi-view results while supporting motion-oriented downstream adaptation.
Problem
Video-model training methods vary widely, and systematic strategies for selecting and curating video data remain underrepresented in the literature.
Method
The paper analyzes text-to-image pretraining, large-scale video pretraining, and high-quality video finetuning, using systematic captioning, filtering, and scaling procedures to curate video data.
Results
The resulting models achieve state-of-the-art text-to-video and image-to-video performance, provide a strong multi-view prior, and outperform specialized novel-view methods.
Takeaways & Limitations
Stable Video Diffusion provides a pretrained motion representation that can be adapted to image-to-video, multi-view synthesis, and motion-specific generation tasks.
Takeaways & Limitations
The approach excels at short video generation but has fundamental shortcomings for long video synthesis.
Abstract
from arXiv · showhide
We present Stable Video Diffusion - a latent video diffusion model for high-resolution, state-of-the-art text-to-video and image-to-video generation. Recently, latent diffusion models trained for 2D image synthesis have been turned into generative video models by inserting temporal layers and finetuning them on small, high-quality video datasets. However, training methods in the literature vary widely, and the field has yet to agree on a unified strategy for curating video data. In this paper, we identify and evaluate three different stages for successful training of video LDMs: text-to-image pretraining, video pretraining, and high-quality video finetuning. Furthermore, we demonstrate the necessity of a well-curated pretraining dataset for generating high-quality videos and present a systematic curation process to train a strong base model, including captioning and filtering strategies. We then explore the impact of finetuning our base model on high-quality data and train a text-to-video model that is competitive with closed-source video generation. We also show that our base model provides a powerful motion representation for downstream tasks such as image-to-video generation and adaptability to camera motion-specific LoRA modules. Finally, we demonstrate that our model provides a strong multi-view 3D-prior and can serve as a base to finetune a multi-view diffusion model that jointly generates multiple views of objects in a feedforward fashion, outperforming image-based methods at a fraction of their compute budget. We release code and model weights at https://github.com/Stability-AI/generative-models .
1. Introduction
The paper argues that video-model performance depends substantially on data selection and training strategy, not only architecture. It introduces a three-stage training approach and a systematic curation workflow, yielding strong text-to-video, image-to-video, motion, and multi-view results.
- The paper identifies data selection as an underrepresented factor in video-model research despite its recognized importance for large-scale training.
- The proposed training strategy separates text-to-image pretraining, large-scale low-resolution video pretraining, and high-resolution finetuning on a smaller high-quality dataset.
- A systematic curation workflow converts a large uncurated video collection into a quality dataset for generative video modeling.
- Pretraining on well-curated datasets produces performance improvements that persist after high-quality finetuning.
- The resulting models achieve state-of-the-art text-to-video and image-to-video performance and outperform prior models according to the paper.
- The pretrained model supports downstream motion control and multi-view generation, outperforming specialized novel-view methods such as Zero123XL and SyncDreamer.
2. Background
Latent video diffusion models extend pretrained image diffusion models with temporal components while operating in a lower-complexity latent space. The paper emphasizes that video data curation has been comparatively underexplored and proposes a systematic curation and three-stage training strategy.
- Latent video diffusion models perform generation in a reduced-complexity latent space.
- Many related models start from pretrained text-to-image models and insert temporal mixing layers into their architectures.
- This work follows an architecture with temporal convolution and attention layers inserted after spatial components, while also using frame-rate micro-conditioning and an EDM framework.
- Data Curation: Video-generation literature has largely lacked systematic data-curation studies, with filtering strategies often introduced ad hoc.
- Data Curation: The paper addresses this gap by systematically studying video curation and introducing a general three-stage training strategy for generative video models.
3. Curating Data for HQ Video Synthesis
The paper develops a staged video-training strategy centered on systematic data processing, annotation, and filtering. Curated pretraining data improves video-model preferences at small and large scales, with gains persisting through high-quality finetuning.
- Training stages: The training strategy separates image pretraining, large-scale video pretraining, and high-quality video finetuning.These are presented as three regimes for generative video modeling.
- Data processing: The processing pipeline detects approximately 4× more clips by identifying cuts beyond those found in metadata.Cut detection is applied at three FPS levels to reduce cuts and fades leaking into synthesized videos.
- Data annotation: The initial Large Video Dataset contains 580M annotated video clip pairs, representing 212 years of content.Clips receive image-based, video-based, and LLM-summarized captions.
- Data curation: Curation filters examples using optical flow, OCR, CLIP, aesthetics, text-image similarity, and synthetic-caption evaluations.For most annotations, the study removes the bottom 12.5%, 25%, or 50% from a 9.8M-example subset and selects thresholds using human-preference Elo rankings.
- Curation results: Models trained on the smaller curated LVD-10M-F subset are preferred in visual quality and prompt-video alignment over models trained on uncurated or competing datasets.The curated subset is four times smaller than WebVid-10M and InternVid-10M, yet its model is preferred by human evaluators in both comparisons.
- Scaling and finetuning: Curated data improves performance at 50M samples, and its advantage remains after 50K steps of high-quality video finetuning.The authors conclude that separating video pretraining from finetuning and using large curated datasets benefits final performance.
4. Training Video Models at Scale
The paper scales Stable Video Diffusion through a pretrained base model and task-specific finetuning, extending it to text-to-video, image-to-video, camera control, frame interpolation, and multi-view generation.
- The authors use an optimal data strategy to train a base model, then finetune it for text-to-video, image-to-video, frame interpolation, and multi-view generation.
- Text-to-Video: The high-resolution text-to-video model is finetuned on approximately 1M high-quality videos containing substantial object motion, steady camera motion, aligned captions, and high visual quality.
- Image-to-Video: The image-to-video models replace text conditioning with CLIP image embeddings and concatenate a noise-augmented conditioning frame across the time axis.
- Image-to-Video: Human voters preferred the 25-frame image-to-video model over GEN-2 and PikaLabs in visual quality comparisons.
- Camera Control: Camera-motion LoRAs are trained for horizontal movement, zooming, and static scenes using small datasets with camera-motion metadata.
- Multi-View Generation: SVD-MV outperforms image-prior and scratch variants and is competitive with state-of-the-art multi-view methods after approximately 12k training iterations.
5. Conclusion
The conclusion presents Stable Video Diffusion as a high-resolution video model built through systematic data curation and staged training. Its learned representation supports image-to-video, camera-control, and multi-view applications.
- Stable Video Diffusion combines systematic data selection and scaling with three separately analyzed stages of video-model training.
- SVD provides a video representation that supports high-quality image-to-video synthesis and camera-control LoRAs.
- SVD acts as a strong 3D prior for multi-view synthesis, achieving state-of-the-art results with a fraction of the compute used by previous methods.
A. Broader Impact and Limitations
The paper situates video diffusion within rapid progress across generative video and multi-view generation. It argues that a video model can provide a useful prior for consistent multi-view synthesis while noting limitations for long videos.
- The approach is limited for long-video synthesis because generating multiple keyframes simultaneously is expensive during both training and inference.
- Earlier video-generation systems commonly used low-resolution or small, noisy datasets, while newer approaches rely on scalable diffusion or autoregressive models and joint image-video training.
- Data curation strategies have been widely used in generative image modeling but remain comparatively underexplored and ad hoc in video generation.
- Multi-view methods often adapt single-image generation models, whereas the paper proposes video models as a better candidate because camera motion forms a video-like structure.
- Temporal attention layers can assist consistent multi-view generation without requiring explicit 3D structures.
C. Data Processing
The curation pipeline converts raw videos into training clips by addressing cuts, static scenes, caption quality, visual quality, and unwanted text. It combines cascaded cut detection, keyframe-aware clipping, motion analysis, synthetic captions, CLIP-based scores, and text detection.
- Motivation: Raw videos are unsuitable for generative video training because they contain cuts, static scenes, misaligned captions, and low-quality examples.These properties can lead models to generate cuts and still scenes, motivating temporal, caption, and visual-quality filtering.
- Cut Detection: A cascade of three cut detectors operating at different frame rates and thresholds detects both sudden scene changes and gradual fades.The approach is designed to capture transitions missed by single-fps detection at one threshold.
- Clipping: Detected cuts are snapped to nearby source keyframes before FFMPEG extracts clips, enabling fast cut-free clipping without inserting new keyframes.The selected keyframe must not cross the detected cut.
- Motion Filtering: Dense optical flow at 2fps is averaged into a global motion score used to filter clips below a minimum-motion threshold.Flow maps are spatially downscaled so their shortest side is 16px, keeping storage tractable.
- Captioning and Quality: The pipeline generates multiple synthetic captions and computes frame-level CLIP text-image similarities and aesthetics scores for filtering.CoCa captures spatial details, VBLIP captures temporal details, and an LLM combines them into captioning alternatives.
- Text Filtering: Text detection annotates written-text bounding boxes, whose total area relative to frame size measures unwanted text for filtering.The pipeline applies text detection to identify clips containing excessive written content.
D.2. Base Model Training and Architecture
The base model is built by adapting Stable Diffusion 2.1 with EDM-style continuous-noise training, temporal layers, and large-scale video pretraining, followed by high-resolution finetuning. The resulting motion representation supports text- and image-conditioned video, interpolation, and camera-motion control.
- Image Pretraining: Stable Diffusion 2.1 is adapted with EDM preconditioning and a noise distribution shifted toward higher noise for high-resolution training.The image backbone is first finetuned for 31k iterations before video training.
- Video Pretraining: Temporal convolution and attention layers are inserted into the image model to create the video model.The resulting model adds 656M temporal parameters and is trained on 14-frame videos at 256 × 384 resolution.
- Conditioning: The base and downstream video models are conditioned on frame rate and motion score, allowing motion variation at inference time.These conditioning signals are used during training for the base, high-resolution text-to-video, and image-to-video models.
- High-Resolution Finetuning: The base model is finetuned on approximately 1M high-quality samples at 576 × 1024 resolution for 50k iterations.The final checkpoint is selected using visual inspection and human evaluation.
- Image-to-Video: Image-to-video finetuning replaces text embeddings with a CLIP image embedding and concatenates a noise-augmented conditioning frame to the UNet input.A frame-axis guidance schedule is used to reduce inconsistency at low guidance and oversaturation at high guidance.
- Motion Control and Interpolation: Camera-motion LoRAs are trained inside temporal attention blocks, while interpolation uses the first and last of five output frames as conditioning frames.The camera-motion LoRAs use rank-16 matrices trained for 5k iterations; the interpolation model converges after about 10k iterations.
E.2.2 Calibrating Filtering Thresholds
Filtering thresholds are calibrated through controlled training runs and human preference comparisons across visual quality and prompt following. The selected thresholds favor cleaner data while balancing these evaluation axes.
- Experimental Setup: Threshold experiments train models with one filtering type varied at a time while keeping architecture and training settings fixed.Models use 8 frames at 256 × 256 resolution and are trained for exactly 40k steps.
- Captioning: CoCa captions have the most beneficial influence, so training samples captions with probabilities pCoCa = 0.5, pV-BLIP = 0.25, and pLLM = 0.25.The sampling strategy preserves caption diversity while favoring CoCa.
- Motion Filtering: Filtering the 25% most static examples is selected because it provides competitive prompt following and quality despite non-filtering ranking higher on prompt following alone.Quality is treated as more important than prompt following for motion-filtering decisions.
- Aesthetics, CLIP, and Text Filtering: The selected thresholds remove the 25% lowest-aesthetics samples, omit 50% under CLIP-score filtering, and remove the 25% largest-text-area samples.The text-area filter ranks highest in both quality and average preference.
- Overall Effect: The combined filtering methods reduce LVD by more than a factor of 3 while producing a much cleaner dataset.The same architecture and hyperparameters are used for the remaining dataset experiments.
E.5.1 Additional Text-to-Video Samples
Additional samples from the text-to-video model introduced in Section 4.2 are presented in Figure 18.
- Figure 18 provides additional samples from the text-to-video model introduced in Section 4.2.
E.5.2 Additional Image-to-Video Samples
The section presents additional samples from the image-to-video model introduced in Section 4.3.
- Additional image-to-video samples illustrate the model introduced in Section 4.3.
E.5.3 Additional Camera Motion LoRA Samples
The section provides additional examples of camera-control motion LoRAs and temporal prompting for motion variation.
- Additional samples show camera-control motion LoRAs tuned as presented in Section 4.3.1.
- Additional text-to-video samples use prompts describing scenes including a hiker, unicorn, snow shoveling, a hen, and a boat near the Eiffel Tower.
- Temporal prompts can vary global motion while preserving spatially similar scenes when the spatial prompt and random seed remain fixed.The spatial prompt describes scene content, while a separate temporal prompt is supplied to temporal cross-attention layers.
E.5.5 Additional Samples on Multi-View Synthesis
The section presents additional examples for image-to-video, camera-motion control, and multi-view generation, including comparisons across multi-view methods and datasets.
- Additional image-to-video samples use the leftmost frame as conditioning.
- Camera-motion LoRA samples correspond to horizontal, static, and zooming motion across the first, second, and third rows.
- Text-to-video samples vary temporal prompts for panning, rotating, and zooming while keeping the spatial scene prompt fixed.
- Additional SVD-MV samples include image-to-multi-view results trained on Objaverse and MVImageNet datasets.
- The multi-view examples include comparisons with other methods and a text-to-image-to-multi-view pipeline using SDXL followed by SVD-MV.