PaperlayerView exact revision on arXiv

Source-linked AI summary

Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis

Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, Sergey Tulyakov

arXiv:2402.14797v1cs.CVcs.AI

TL;DR

Image-derived video generators face the challenge of video redundancy and the computational cost of processing every frame. Snap Video adapts EDM and introduces compressed spatiotemporal FIT transformers; it reports faster generation, benchmark performance, and strong user-study preferences.

  • Problem

    Image-based diffusion frameworks do not account for video redundancy, and U-Nets process every frame, adding computational overhead that limits scalability.

    Source: §1 · p. 2

  • Method

    Snap Video adapts EDM for high-resolution video and uses FIT transformers that jointly model spatial and temporal dimensions in a compressed representation.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    Snap Video reports state-of-the-art benchmark performance, with FIT training 3.31× faster and inference 4.49× faster than U-Nets.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The authors conclude that compressed joint spatiotemporal modeling improves temporal consistency and motion modeling while supporting FIT scaling to billions of parameters.

    Source: §5 · p. 8

    The generator often spells text incorrectly and may miss requested object counts, which the authors relate to challenges in captioned video data.

    Source: §F.1 · p. 25

Abstract

from arXiv · show

Contemporary models for generating images show remarkable quality and versatility. Swayed by these advantages, the research community repurposes them to generate videos. Since video content is highly redundant, we argue that naively bringing advances of image models to the video generation domain reduces motion fidelity, visual quality and impairs scalability. In this work, we build Snap Video, a video-first model that systematically addresses these challenges. To do that, we first extend the EDM framework to take into account spatially and temporally redundant pixels and naturally support video generation. Second, we show that a U-Net - a workhorse behind image generation - scales poorly when generating videos, requiring significant computational overhead. Hence, we propose a new transformer-based architecture that trains 3.31 times faster than U-Nets (and is ~4.5 faster at inference). This allows us to efficiently train a text-to-video model with billions of parameters for the first time, reach state-of-the-art results on a number of benchmarks, and generate videos with substantially higher quality, temporal consistency, and motion complexity. The user studies showed that our model was favored by a large margin over the most recent methods. See our website at https://snap-research.github.io/snapvideo/.

1. Introduction

Snap Video argues that image-first video generation overlooks video redundancy and scales poorly, then introduces video-specific diffusion training and compressed spatiotemporal transformers. It reports strong benchmark performance, faster generation, and favorable user-study comparisons.

1. Introduction

  • Large text-to-image models made high-quality image creation and editing accessible beyond professionals, alongside rapid progress in large-scale text-to-video models.The passage describes high-quality image generation and editing becoming accessible to everyone.Source: §1 · p. 1
  • Existing large-scale video generators build on image architectures such as U-Nets, adding temporal layers and often training on both videos and separate images.The paper instead reformulates EDM for high-resolution videos and treats images as high-frame-rate videos during joint training.Source: §1 · p. 2
  • Image-first video generation overlooks redundancy between successive frames, while processing every frame with U-Nets raises computational overhead and limits scalability.The paper links these challenges to motion artifacts, dynamic images, and reduced motion fidelity.Source: §1 · p. 2
  • The compressed FIT transformer reduces training time 3.31× and inference time 4.49× versus U-Nets while jointly modeling spatial and temporal dimensions.It represents those dimensions as a single compressed 1D latent vector.Source: §1 · p. 2
  • On UCF101 and MSR-VTT, Snap Video reports state-of-the-art performance across benchmarks; participants preferred it for prompt alignment in 81% of cases versus Gen-2 [11].Prompt-alignment preference was 80% versus Pika and 81% versus Floor33 [17].Source: §1 · p. 2

2. Related Work

Related work spans text-to-video generation, high-resolution diffusion strategies, and diffusion frameworks. Snap Video distinguishes itself from image-derived video architectures with a compressed representation for joint spatiotemporal modeling.

Video Generation

  • Prior text-to-video work includes autoregressive models and GANs, followed by diffusion systems that adapt image-generation methods to video.Examples include cascaded spatial and temporal upsamplers, correlated noise, latent diffusion, and motion modules added to pretrained generators.Source: §2 · p. 2
  • Many diffusion video systems use U-Nets with separable spatial and temporal computation, which the paper says limits motion modeling.VideoFactory proposes Swapped Spatiotemporal Cross-Attention to improve interactions across spatial and temporal modalities.Source: §2 · p. 2
  • Snap Video uses FIT transformers and a learnable compressed video representation for training savings, scalability, and improved temporal consistency and motion modeling.Its global joint spatiotemporal modeling strategy is presented as a contrast to adapting U-Nets for video.Source: §2 · p. 2

High-Resolution Generation

  • High-resolution generation approaches include cascaded diffusion, latent diffusion, and end-to-end methods that adapt noise schedules or progressively denoise at higher resolutions.The paper adopts a two-stage cascade to avoid temporal flicker associated with latent autoencoders and to specialize low- and high-resolution models.Source: §2 · p. 3

Diffusion Frameworks

  • The paper adopts EDM's score-based generative modeling framework and reformulates it for high-resolution video generation.The passage describes diffusion as forward noise addition followed by reverse denoising.Source: §2 · p. 3

3. Method

Snap Video adapts EDM for high-resolution video and develops a scalable transformer that jointly models compressed spatial and temporal information. The method also aligns image-video training and specifies training and inference procedures.

3. Method

  • The method extends EDM to high-dimensional video inputs and uses a FIT-based transformer designed to scale to billions of parameters and tens of thousands of patches.The method overview also covers image-video modality matching, training, and inference.Source: §3 · p. 3

3.2. EDM for High-Resolution Video Generation

  • Averaging redundant video pixels can raise SNR above the image setting, so the method scales inputs to preserve the original-resolution SNR.The forward process uses x/σin, and the framework is equivalent to EDM when σin = 1 while avoiding an unstable term.Source: §3.2 · pp. 3–4, Table 1 · p. 4

3.3. Image-Video Modality Matching

  • Images are treated as videos with infinite frame-rate, and variable-frame-rate training blends the image and video modalities under one diffusion process.The passage notes that separate input-scaling factors would increase framework complexity and that image training would not foster temporal reasoning.Source: §3.3 · p. 4

3.4. Scalable Video Generator

  • Snap Video’s FIT architecture jointly models space and time on a compressed video representation, unlike U-Nets that repeat spatial computation per frame.FIT blocks read patch information into latent tokens, compute there, and write results back; design choices include spatial-only patches, groups spanning all frames, more latent tokens, and replacing costly local attention with feed-forward modules.Source: Figure 3 · p. 5, §3.4 · p. 5
  • The model uses T5-11B text embeddings and tokens for noise level, frame-rate, and resolution, with a two-stage cascade upsampling 36×64px videos to 288 × 512px.The second-stage model is trained with variable noise on its low-resolution inputs.Source: §3.4 · p. 6

3.5. Training

  • Training uses LAMB with a 5e−3 learning rate and batch sizes of 2048 videos and 2048 images; the first stage runs 550k steps and the second-stage finetuning runs 370k iterations.For 16-frame videos, σin is 4 for the first stage and 32 for the second stage.Source: §3.5 · p. 6

3.6. Inference

  • Inference uses a deterministic sampler with 256 first-stage and 40 second-stage sampling steps, plus classifier-free guidance unless otherwise specified.Dynamic thresholding and oscillating guidance consistently improve sample quality.Source: §3.6 · p. 6

4. Evaluation

The evaluation tests Snap Video’s architecture and diffusion choices through ablations, then compares it with video-generation baselines on benchmark datasets and in user studies. The reported comparisons favor Snap Video on video-quality and motion criteria, while qualitative examples highlight temporal coherence.

4.1. Datasets; 4.2. Evaluation Protocol

  • Zero-shot evaluation uses UCF101 and MSR-VTT, while the protocol generates 16-frame videos at 512 × 288px and evaluates benchmark-specific metrics and prompts.A separate user study assesses photorealism, video-text alignment, and motion quantity and quality.Source: §4.1 · p. 6, §4.2 · p. 7

4.3. Ablations

  • On the internal dataset, a 500M-parameter FIT trains 3.31× faster and runs inference 4.49× faster than a 284M-parameter U-Net, while surpassing it on FID and CLIPSIM.The 3.9B-parameter FIT increases inference time only 1.24× relative to the 284M U-Net.Source: Table 2 · p. 6, §4.3 · p. 7
  • The modified diffusion framework improves over EDM across all metrics; setting σdata = 1 and treating images as infinite-frame-rate videos benefit performance, while σin < sT impairs it.The authors attribute the σdata = 1 benefit to a training target and loss weight matching v-prediction.Source: §4.3 · p. 7

4.4. Quantitative Evaluation

  • Snap Video improves FID and FVD over baselines on UCF101 and MSR-VTT, though its MSR-VTT CLIPSIM is lower than Make-A-Video’s.The authors attribute the CLIPSIM difference to using T5 rather than CLIP text embeddings.Source: §4.4 · p. 7, Table 4 · p. 7, Table 5 · p. 7
  • In a user study, Snap Video outperforms the compared generators on video-text alignment and motion quantity and quality, with photorealism comparable to Gen-2.The study compares Gen-2, PikaLabs, and Floor33 on 65 prompts describing vivid motion.Source: §4.4 · p. 8, Table 6 · p. 8

4.5. Qualitative Evaluation

  • Qualitative comparisons report more photorealistic, text-aligned samples with vivid, high-quality motion and fewer flickering artifacts than the compared generators.The comparison uses publicly released samples, which may have been selected to highlight baseline strengths.Source: §4.5 · p. 8, Figure 4 · p. 8

5. Conclusions

Snap Video treats videos as first-class citizens by modifying EDM and replacing U-Nets with scalable FIT transformers. It reports state-of-the-art results on UCF101, MSR-VTT, and user studies, particularly for motion quality.

5. Conclusions

  • Snap Video modifies EDM for high-resolution video generation and treats images as high-frame-rate videos to avoid image-video modality mismatch.It replaces U-Nets with FIT transformers scaled to billions of parameters, using a learnable compressed video representation for joint spatiotemporal modeling.Source: §5 · p. 8

Supplementary Material

The supplementary material identifies the architectural details of the U-Net baselines.

Supplementary Material

  • Table 7 provides the architectural details of the U-Net baselines.Source: Table 7 · p. 12

A. Architecture details

Snap Video FIT modifies vanilla FITs to improve temporal modeling and handle high-resolution video inputs.

A. Architecture details

  • FIT patches span one frame, patch groups span the video temporally, and 4 × 4px spatial patches preserve spatial modeling; the configuration yields 147,456 input patches at high resolution.The model replaces local attention with feedforward operations after cross-attention and uses 768 latent tokens to store additional temporal information.Source: A. Architecture details · p. 12

B. Training details

The training details and additional evaluations describe model settings, sampling behavior, user-study comparisons, and qualitative results against other video generators. The reported user studies favor Snap Video on photorealism and motion quality, with video-text alignment higher than all compared methods except Imagen Video.

B. Training details

  • Snap Video training uses LAMB at a 5e−3 learning rate for 550k steps, with 10k warmup steps and a batch of 2048 videos plus 2048 images.The largest model also uses gradient clipping, dropout, and exponential moving averages; experiments run on Nvidia A100 GPUs.Source: B. Training details · p. 12, Table 9 · p. 12

C.1. Sampler Parameters Ablations

  • Classifier-free guidance can improve FVD and CLIPSIM, but high guidance scales increase saturation; dynamic thresholding and oscillating guidance reduce that effect.FVD improves through 256 sampling steps, while high-quality samples are already produced at 64 steps.Source: §C.1 · p. 13, Figure 5 · p. 13, Figure 6 · p. 13

C.2. User Studies

  • User studies found increased photorealism, alignment higher than all baselines except Imagen Video, and consistently better motion quality than Make-A-Video, PYoCo, Video LDM, and Imagen Video.Participants cast five votes per sample; the table reports the percentage of votes favoring Snap Video.Source: Table 10 · p. 13, §C.2 · p. 13

C.3. Qualitative Results Against Baselines

  • Against Make-A-Video, PYoCo, Video LDM, and Imagen Video, Snap Video preserves natural, temporally consistent motion in large-motion scenes where baselines often flicker or show inconsistent objects.Qualitative comparisons also report better prompt alignment and large amounts of coherent motion versus Gen-2, PikaLab, and Floor33.Source: §C.3 · p. 13

C.4. Additional Qualitative Results

Additional qualitative evaluations show diverse outputs across complex prompts, plausible novel object views, and zero-shot UCF101 samples. A hierarchical strategy also generates longer, higher-framerate videos by conditioning on previously generated frames.

C.4. Additional Qualitative Results

  • Complex prompts yield varied concepts and challenging motion—including large camera movement, POV, and fast-moving objects—while maintaining temporal consistency and avoiding flicker.The authors attribute these motion capabilities to joint spatiotemporal modeling.Source: §C.4 · pp. 13–14
  • Circular, orbiting, or moving-camera prompts produce plausible novel object views, which the authors say suggest an understanding of modeled objects’ 3D geometry.The same section reports diverse samples for a given prompt.Source: §C.4 · p. 14

C.5. Hierarchical Video Generation

  • Hierarchical generation autoregressively creates a video at the lowest framerate, then increases framerate while conditioning on frames generated at lower rates.The method uses the last frame as conditioning at the first level and all previously generated lower-framerate frames at subsequent levels.Source: §C.5 · p. 14

C.6. Zero-Shot UCF101 Evaluation

  • For zero-shot UCF101, the model is conditioned to produce low-resolution videos to better match the dataset’s low-resolution YouTube-video distribution.Class labels are converted to descriptive prompts because labels are not always sufficiently descriptive.Source: §C.6 · p. 14

D. Derivation of EDM Denoising Objective

The derivation rewrites the EDM denoising objective in terms of Fθ after modifying the forward process with an input-scaling factor. It also connects the resulting objective to v-prediction under a stated condition.

D. Derivation of EDM Denoising Objective

  • The derivation expresses the EDM denoising objective in terms of Fθ after introducing input scaling σin into the forward process.It begins from the original objective for Dθ and substitutes the denoiser and EDM coefficient definitions.Source: D. Derivation of EDM Denoising Objective · p. 22
  • The derivation identifies the training target as a rescaled, negated v-prediction objective and the loss weight as a reweighted 1 + SNR weighting.The passage cites [45] for the 1 + SNR v-prediction weighting.Source: D. Derivation of EDM Denoising Objective · p. 22
  • When σdata = 1, EDM is equivalent to the v-prediction formulation.The derivation uses this observation when rewriting the framework to avoid a spurious term.Source: D. Derivation of EDM Denoising Objective · p. 22

E. Derivation of Our Diffusion Framework

The framework derivation chooses a target that preserves the original EDM denoising target across input-scaling values. It also retains EDM’s loss weighting while allowing control over input signal quantity.

E. Derivation of Our Diffusion Framework

  • The proposed framework keeps Ftgt equal to the original EDM target for every σin, so Fθ learns the same denoising function while input signal quantity remains adjustable.The derivation imposes this target and retains EDM’s loss weight λ(σ).Source: E. Derivation of Our Diffusion Framework · pp. 23–24
  • The framework preserves the original loss weighting across noise levels, so adjusting input scaling does not alter how those levels are weighted.The derivation separately constructs normalization and input-scaling components.Source: E. Derivation of Our Diffusion Framework · pp. 23–24

F. Discussion

  • The discussion section covers framework limitations and societal impact.The passage points readers to Sec. F.1 and Sec. F.2, respectively.Source: F. Discussion · p. 25

F.1. Limitations

Snap Video has limitations in prompt fidelity, motion-heavy video compression, and output resolution. The authors also identify misuse and dataset-bias risks, and describe safeguards they plan to add.

F.1. Limitations

  • Text may be misspelled, requested object counts or complex spatial relations may be wrong, negation may be ignored, and stylization may be absent or unanimated.The authors attribute these failures to data and learning challenges, including noisy descriptions and limited high-aesthetic training data.Source: §F.1 · p. 25
  • Very high motion can produce visible patch artifacts when video content is compressed into a fixed-size latent representation.Source: §F.1 · p. 25
  • The cascade currently generates 512 × 288px video; the authors expect improvements at that resolution to carry to higher resolutions given an upsampler.The stages are independently trained and agnostic to the previous-stage generator; integrating additional cascade stages is future work.Source: §F.1 · p. 25

F.2. Societal Impact

  • The authors cite risks of deceptive outputs and training-data biases, so they withhold public access and plan data cleaning, prompt and output filtering, and watermarking.Source: §F.2 · p. 25

Related papers

Loading 2402.14797v1…