VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation

1. Introduction

• Text-to-video generation must produce high-quality visual content and temporally smooth, realistic motion matching the text, while large-scale text-video datasets are hard to find.Source: §1 · p. 1
• VideoGen generates a high-quality image from the prompt as a reference, then conditions cascaded latent video diffusion on that image and the text.A flow-based scheme can temporally upsample the latent sequence before a decoder maps it to video.Source: §1 · p. 1
• The reference image improves visual fidelity and frees the diffusion model from learning visual content, letting it focus on video dynamics.The paper says this approach is more training-efficient than Imagen Video's use of image-text pairs for joint training.Source: §1 · p. 1
• The video decoder needs only latent representations, enabling training on easily available unlabeled videos rather than only video-text pairs.The paper says high-quality unpaired videos improve motion smoothness and realism.Source: §1 · pp. 1–2
• An off-the-shelf text-to-image model generates the reference image to improve frame content quality.Source: §1 · p. 2
• VideoGen reports state-of-the-art results in both quantitative and qualitative measures.The authors attribute improvements to a high-quality reference image and training the decoder on unlabeled video data.Source: §1 · pp. 2–3

2. Related Work

• Earlier text-to-video approaches include recurrent-attention VAEs, GAN extensions, and systems such as GODIVA, NÜWA, and CogVideo.Source: §2 · p. 3
• More recent work includes fine-tuning methods and video diffusion systems, including Make-A-Video, MagicVideo, Video Diffusion Model, Imagen Video, and latent video diffusion models.Source: §2 · p. 3
• Unlike Make-A-Video [46], VideoGen uses an image as a reference to guide video content generation rather than using its embedding to replace the text embedding for animation.Source: §2 · p. 3
• VideoGen's autoencoder image representation retains semantics and reconstruction details, whereas Make-A-Video's CLIP image embedding is primarily semantic.The authors connect this distinction to Make-A-Video's blurrier results.Source: §2 · p. 3
• VideoGen combines latent diffusion for more efficient sampling with reference guidance and a decoder that can be trained using high-quality unpaired videos.The comparison for sampling efficiency is with Make-A-Video [46] and Imagen Video [19].Source: §2 · p. 3

3. Approach

• The inference pipeline generates a reference image from the text prompt, encodes image and text conditions, produces latent video representations, temporally upsamples them, and decodes them into video.Source: §3 · p. 3, Figure 2 · p. 4

3.1. Reference Image Generation

• A pretrained text-to-image model generates the reference image, while training uses the video's first frame as the reference.The image is used without additional processing.Source: §3.1 · p. 3

3.2. Reference-Guided Latent Video Diffusion

• Cascaded latent video diffusion first generates 16 frames of 16 × 16 representations, then two diffusion networks raise spatial resolution to 32 × 32 and 64 × 64.Source: §3.2 · p. 3
• The diffusion networks use text cross-attention and a reference-image representation concatenated with the first-frame input representation.The reference image is encoded at resolutions corresponding to the diffusion networks.Source: §3.2 · p. 4

3.3. Flow-based Temporal Super-resolution

• Temporal super-resolution estimates motion flow in latent space, warps representations, and refines them with a denoising diffusion network.Combining low-frequency warped content with high-frequency refined content is repeated three times to produce 8× upsampled representations.Source: §3.3 · p. 4

3.4. Video Decoder

• The video decoder adapts the Stable Diffusion 8× image decoder with temporal convolutions and attention following the spatial operations.This allows initialization from the pretrained image decoder's convolution and spatial-attention parameters.Source: §3.4 · p. 4

3.5. Training

• The three diffusion networks are trained on video-text pairs, using each video's first frame as the reference; pretrained text, image, and text-to-image models remain frozen.The diffusion networks are initialized using pretrained image-model parameters.Source: §3.5 · pp. 4–5
• The flow-based super-resolution network and decoder are trained on unpaired videos, with the decoder learning to map degraded-video latents to the original video.The training set contains 40K YouTube clips of 100 frames; flow targets use high-FPS video representations.Source: §3.5 · p. 5

4.1. Datasets and Metrics

• Training used WebVid-10M for latent video diffusion and 40,000 clips from over 2,000 4K, 60-FPS videos for temporal super-resolution and decoder training.Evaluation used UCF-101 and MSR-VTT; generated videos were limited to 16×256×256 because the evaluation encoders do not expect higher resolution or frame rate.Source: §4.1 · pp. 5–6

4.2. Results

• From 33 to 71.6, VideoGen improves zero-shot UCF-101 IS over the second-best score; it also leads both UCF-101 settings and MSR-VTT CLIPSIM without fine-tuning.The MSR-VTT comparison uses average CLIPSIM scores to evaluate text-video alignment.Source: Table 1 · p. 6, §4.2 · p. 7, Table 2 · p. 7
• VideoGen is second-best in zero-shot FVD; qualitatively, its lake details are clearer than Make-A-Video’s and its frames are less blurry than 1280×768 Imagen Video’s.The authors suggest training-data distribution differences may explain the FVD result and note that fine-tuning targets general text-to-video generation, not this setting.Source: Figure 4 · p. 6, §4.2 · p. 7

4.3. Ablation Study

• Removing reference guidance lowers IS and CLIPSIM; with guidance, frames show more texture detail in dark clouds and grass.The ablation also reports that the same text prompt can produce different videos from different reference images.Source: §4.3 · pp. 7–8, Figure 5 · p. 7, Table 3 · p. 8
• Motion-flow-guided temporal super-resolution produces more stable, continuous interpolated frames than spherical interpolation, which shows artifacts in the examples.The reported examples include broken fire and artifacts on a shoe without flow guidance.Source: Figure 6 · p. 7, §4.3 · p. 8
• The video decoder restores sharper texture than Stable Diffusion’s original image decoder and produces temporally smoother videos.The authors attribute the decoder’s enhanced effect to training with input degradations.Source: §4.3 · p. 8, Figure 7 · p. 8

4.4. User Study

• 90% of users preferred VideoGen to Make-A-Video for video quality, and 76% preferred it to ImagenVideo; VideoGen also outperformed both on text-video alignment.The study used 30 prompts and 17 annotators.Source: §4.4 · p. 8, Figure 8 · p. 8

• A high-quality reference image from a text-to-image system improves the generated video’s visual fidelity.Source: §5 · p. 8
• Using the reference image to guide latent video diffusion lets the diffusion model focus more on learning motion.Source: §5 · p. 8
• The decoder can be trained on high-quality unpaired videos because it does not depend on video-text pairs.Source: §5 · p. 8