PaperlayerView exact revision on arXiv

Source-linked AI summary

Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation

Jiaxi Gu, Shicong Wang, Haoyu Zhao, Tianyi Lu, Xing Zhang, Zuxuan Wu, Songcen Xu, Wei Zhang, Yu-Gang Jiang, Hang Xu

arXiv:2309.03549v1cs.CVcs.AIcs.MM

TL;DR

Text-to-video LDMs face compute and memory limits that restrict frame counts, while prediction-based extensions can add training cost and jittering. VidRD reuses latent features to iteratively extend an initial clip and reports strong quantitative and qualitative results.

  • Problem

    A typical LDM generates only a few video frames under computation and memory constraints, while maintaining temporal consistency with diverse content remains challenging.

    Source: §1 · p. 2

  • Method

    VidRD iteratively extends an initial clip by reusing latent features and the previous diffusion process, alongside temporal decoder layers and video-text data strategies.

    Source: §1 · p. 2

  • Results

    VidRD reports strong quantitative and qualitative performance, including UCF-101 FVD of 363.19 and IS of 39.37.

    Source: §1 · p. 2, §6 · p. 13

  • Takeaways & Limitations

    The paper presents iterative latent reuse and temporal decoder adaptation as an approach for generating smoother, longer text-guided videos.

    Source: §6 · p. 13

Abstract

from arXiv · show

Inspired by the remarkable success of Latent Diffusion Models (LDMs) for image synthesis, we study LDM for text-to-video generation, which is a formidable challenge due to the computational and memory constraints during both model training and inference. A single LDM is usually only capable of generating a very limited number of video frames. Some existing works focus on separate prediction models for generating more video frames, which suffer from additional training cost and frame-level jittering, however. In this paper, we propose a framework called "Reuse and Diffuse" dubbed $\textit{VidRD}$ to produce more frames following the frames already generated by an LDM. Conditioned on an initial video clip with a small number of frames, additional frames are iteratively generated by reusing the original latent features and following the previous diffusion process. Besides, for the autoencoder used for translation between pixel space and latent space, we inject temporal layers into its decoder and fine-tune these layers for higher temporal consistency. We also propose a set of strategies for composing video-text data that involve diverse content from multiple existing datasets including video datasets for action recognition and image-text datasets. Extensive experiments show that our method achieves good results in both quantitative and qualitative evaluations. Our project page is available $\href{https://anonymous0x233.github.io/ReuseAndDiffuse/}{here}$.

1 INTRODUCTION

VidRD targets limited frame counts and temporal consistency in text-to-video generation by iteratively reusing latent features and adapting a pretrained LDM. It also proposes video-text data strategies and reports strong UCF-101 results.

1 INTRODUCTION

  • A typical LDM generates few frames under computation and memory constraints, while temporal consistency and diverse content remain challenges.Separate prediction models add training cost and inference time, and frame-level jittering remains difficult to solve.Source: §1 · p. 2
  • VidRD reports UCF-101 FVD of 363.19 and IS of 39.37, with experiments described as performing well against existing methods.Qualitative evaluations are also reported as good.Source: §1 · p. 2
  • VidRD generates subsequent frames iteratively by reusing latent features from an initial clip and imitating the prior diffusion process.Its temporal-aware LDM adapts a pretrained image LDM and adds temporal decoder layers; FNR, PNS, and DSG support iterative generation and consistency.Source: §1 · p. 2
  • The proposed data strategies use LLMs to segment and caption action-recognition videos and transform image-text data into pseudo-videos with random zooming and panning.The strategies address the scarcity of high-quality video-text data.Source: §1 · p. 2

2 RELATED WORK

Related work traces text-guided video synthesis from earlier GAN and transformer approaches to diffusion-based methods. It also discusses the challenges of extending generation to longer videos.

Image synthesis models

  • Diffusion models advanced text-guided image synthesis, following GAN and autoregressive approaches such as DALL-E and Parti.The passage notes high computation costs and sequential error accumulation for autoregressive methods.Source: §2 · p. 3

Video synthesis models

  • Video diffusion methods extend image LDMs with temporal layers or video noise priors, while Video LDM also uses interpolation and upsampling models.Video LDM fine-tunes added temporal layers with video data; PYoCo designs a video noise prior for temporal consistency.Source: §2 · p. 3

Iterative video generation

  • Longer-video approaches include TATS, separate prediction models, and unified interpolation-and-prediction training, each with distinct costs or training challenges.Separate prediction multiplies training cost, while LVDM's unified training makes training more difficult and unstable.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe diffusion as iterative noising followed by learned denoising, and explain how latent diffusion uses an autoencoder to reduce computation and memory.

3.1 LATENT DIFFUSION MODELS

  • The forward diffusion process adds noise over time, and the schedule permits direct sampling of x_t from x_0 at a chosen diffusion time.The variance-preserving schedule uses a cumulative product of α_i to define ᾱ_t.Source: §3.1 · p. 4
  • A learned model predicts noise to iteratively recover x_0 from x_T, which approaches Gaussian noise after sufficiently many forward steps.Generation can use deterministic DDIM sampling from Gaussian noise after training.Source: §3.1 · p. 4
  • Latent diffusion compresses pixels into latent features with an encoder and decodes generated latents back into pixels, reducing computation and memory demands.Image-synthesis autoencoders operate on static images, and existing video autoencoders process frames individually without temporal information.Source: §3.1 · p. 4

4 METHOD

VidRD adapts an image LDM for video with temporal modules and a decoder fine-tuned for temporal consistency. It combines diverse data-preparation strategies with iterative clip generation that reuses prior latent features and noise.

4.1 MODEL ARCHITECTURE

  • The video LDM inherits Stable Diffusion weights, adds 3D-convolution Temp-Conv and temporal-attention Temp-Attn modules, and trains spatial attention and temporal modules using video data.The text encoder is frozen; the autoencoder decoder also receives trainable temporal layers, and image-text examples are converted into pseudo-videos for unified training.Source: §4.1 · pp. 4, 6

4.2 VIDEO-TEXT DATA COMPOSITION

  • VidRD composes training data from image-text, short-video, and long-video classification datasets, adapting each source into captioned video examples.Images become pseudo-videos through random zooming and panning; short clips use a caption generated from a randomly selected frame, while long videos are filtered, segmented, and captioned.Source: §4.2 · pp. 6–7

4.3 LONGER VIDEO GENERATION

  • VidRD generates longer videos with one LDM by reusing prior-clip features and reversed noise; PNS adds randomness, while DSG reuses denoising latents to guide continuity without simply duplicating frames.FNR alone can lead to repetitive content cycling; PNS changes noise on non-prompt frames, and DSG guidance weakens as frames approach the prompt-frame boundary.Source: §4.3 · pp. 8–9

4.4 TEMPORAL-AWARE DECODER FINE-TUNING

  • The autoencoder keeps its encoder unchanged and frozen, while video-data fine-tuning trains newly added 3D-convolution Temp-Conv layers in the decoder.The added layers are zero-initialized with residual connections; fine-tuning uses the U-Net training datasets and an adversarial loss setup.Source: §4.4 · p. 9

5 EXPERIMENTS

VidRD performs strongly on zero-shot UCF-101 evaluation, and the experiments examine its data and inference choices. The study also reports qualitative comparisons and trade-offs among ablation settings.

5.1 EXPERIMENTAL SETUPS

  • Training uses Stable Diffusion v2.1 to initialize spatial layers, freezes the VAE and text encoder, and trains newly added temporal layers and spatial transformer blocks.DDIM is used for LDM sampling, and only newly added temporal layers in the VAE decoder are trainable.Source: §5.1 · p. 10
  • VidRD training combines four data types: well-captioned videos, short and long video-classification datasets, and image-text data converted into pseudo-videos.The described image-derived set contains 640K pseudo-videos averaging 2 seconds; the long-video strategy produces 800K captioned videos averaging 5 seconds.Source: §5.1 · p. 10, Table 1 · p. 11
  • For quantitative evaluation, VidRD uses FVD calculated with a trained I3D model and video IS calculated with a trained C3D model; generated videos are 16 frames at 256 × 256 unless otherwise specified.Source: §5.1 · p. 11, §5.2 · p. 11

5.2 MAIN RESULTS

  • VidRD achieves better FVD and IS than other methods on zero-shot UCF-101 while using fewer training videos.The evaluation generates 10K videos following the training set’s class distribution; the authors report that fine-tuning the VAE decoder improves VidRD further.Source: §5.2 · p. 11, Table 2 · p. 11
  • Same-prompt examples show VidRD performing well in video structure and appearance compared with Make-A-Video and Imagen Video.The authors note that current video-generation metrics may not fully reflect perception, motivating qualitative evaluation.Source: §5.2 · p. 12, Figure 7 · p. 12

5.3 ABLATION STUDY

  • The base-model ablation selects 50 inference steps for efficiency and guidance scale w = 10.0 to balance FVD and IS.The number of inference steps has relatively small impact; smaller guidance scales can lower FVD but also lower IS.Source: §5.3 · p. 12, Figure 8 · p. 13
  • Pseudo-videos made by zooming and panning static images improve temporal consistency but compromise visual appearance relative to static-image training.The authors attribute the difference to pseudo-videos providing more training for temporal layers, while static images focus more on spatial layers.Source: §5.3 · p. 12, Table 3 · p. 13
  • Inference ablations find that more prompt frames can improve metrics but increase content cycling, while reusing noise and denoising guidance improve temporal consistency.Too-large β can also promote content cycling not measured by FVD or IS; the authors set β = 0.4 as a balance, and Figure 10 illustrates abrupt changes when α or β is zero.Source: §5.3 · p. 13

6 CONCLUSION

The conclusion presents VidRD as a text-guided video framework combining iterative generation, temporal decoder layers, and strategies for using diverse existing datasets. The authors report strong quantitative and qualitative evaluations.

6 CONCLUSION

  • VidRD generates longer videos iteratively by reusing noise and imitating the diffusion process clip by clip.The framework is presented as a method for generating smooth videos with text guidance.Source: §6 · p. 13
  • The framework uses temporal layers in the autoencoder and strategies for training with action-recognition video and image-text datasets.Source: §6 · p. 13
  • The authors report that VidRD excels in both quantitative and qualitative evaluations.Source: §6 · p. 13

Related papers

Loading 2309.03549v1…