PaperlayerView exact revision on arXiv

Source-linked AI summary

FlexiFilm: Long Video Generation with Flexible Conditions

Yichen Ouyang, jianhao Yuan, Hao Zhao, Gaoang Wang, Bo zhao

arXiv:2404.18620v1cs.CV

TL;DR

Long-video generation with diffusion models is challenged by limited conditioning and sampling strategies. FlexiFilm combines temporal conditioning, co-training, and resampling; the paper reports stronger qualitative and quantitative performance and generation of clips lasting 200+ frames.

  • Problem

    Diffusion video models adapted from image generation often degrade on long videos, where insufficient temporal conditioning and sampling strategies are associated with inconsistency and overexposure.

    Source: §1 · p. 1

  • Method

    FlexiFilm combines a multimodal temporal conditioner, co-training of temporal modules, and resampling for multi-round generation.

    Source: §1 · p. 1

  • Results

    FlexiFilm demonstrates superior qualitative and quantitative performance over existing methods and generates video clips lasting 200+ frames with correct temporal and semantic relationships to a reference image.

    Source: §4.2.1 · p. 5, §6 · p. 9

  • Takeaways & Limitations

    The framework is reported to enable much longer recursive video generation without harming quality.

    Source: §1 · p. 1

Abstract

from arXiv · show

Generating long and consistent videos has emerged as a significant yet challenging problem. While most existing diffusion-based video generation models, derived from image generation models, demonstrate promising performance in generating short videos, their simple conditioning mechanism and sampling strategy-originally designed for image generation-cause severe performance degradation when adapted to long video generation. This results in prominent temporal inconsistency and overexposure. Thus, in this work, we introduce FlexiFilm, a new diffusion model tailored for long video generation. Our framework incorporates a temporal conditioner to establish a more consistent relationship between generation and multi-modal conditions, and a resampling strategy to tackle overexposure. Empirical results demonstrate FlexiFilm generates long and consistent videos, each over 30 seconds in length, outperforming competitors in qualitative and quantitative analyses. Project page: https://y-ichen.github.io/FlexiFilm-Page/

1 INTRODUCTION

Diffusion video models often degrade on long videos, with temporal inconsistency linked to insufficient conditioning and overexposure linked to non-zero-SNR sampling. FlexiFilm combines temporal conditioning, co-training, and resampling to address these challenges and generate longer, more consistent videos.

1 INTRODUCTION

  • Existing diffusion video models are usually limited to a few seconds and degrade on long videos, where temporal inconsistency and overexposure are prominent failures.Insufficient temporal information in conditioning is linked to inconsistency; a scheduler that fails to reach zero SNR is linked to overexposure or structural collapse, especially in recursive generation.Source: §1 · p. 1
  • FlexiFilm combines a temporal conditioner, co-training, and resampling to support controllable, high-quality long-video generation.The conditioner extends guidance across generated frames, co-training promotes collaboration between temporal modules, and resampling addresses the non-zero-SNR problem during multi-round inference.Source: §1 · p. 1
  • FlexiFilm is reported to generate longer, more consistent videos than baseline methods in visual quality and evaluation metrics.The reported experiments attribute the framework’s approach to addressing both temporal inconsistency and overexposure.Source: §1 · p. 1

2 RELATED WORKS

Prior work includes autoregressive video generation and diffusion models adapted from image generation, alongside approaches that extend videos through keyframes, noise rescheduling, or overlapping clips. The paper presents a video projector that combines spatial features from individual condition frames with temporal information across frames.

2 RELATED WORKS

  • Autoregressive video models generate subsequent frames with decoder-only transformers, but their large-scale training requirements are often unrealistic for common researchers [21].They use visual and text tokenizers to represent video frames as discrete tokens.Source: §2 · p. 1
  • The video projector processes condition frames separately for spatial information, then together through temporal transformers to produce features with spatial and temporal information.The framework’s temporal conditioner can process a single image or multi-frame video for long-video guidance.Source: Figure 1 · p. 2, §2 · p. 2
  • Diffusion video models adapted from image models usually generate only short videos, with simple conditioning and sampling strategies linked to long-video inconsistency and overexposure.Examples include VideoCrafter [7], which adds temporal layers to a 3D U-Net, and AnimateDiff [10], which adds a motion module to a pretrained text-to-image model.Source: §2 · p. 2
  • Existing long-video approaches use keyframe generation followed by interpolation, rescheduled noise with temporal attention, or parallel inference over overlapping clips [8] [37] [42] [25] [32].The paper says these approaches struggle to balance video quality and consistency.Source: §2 · p. 2

3.1 Overview

The FlexiFilm overview organizes the method into model architecture, temporal-module co-training, resampling for multi-round generation, and construction of a long-video dataset.

3.1 Overview

  • The overview introduces the model architecture and temporal conditioner for multimodal guidance that models relationships between long-term generations and their conditions.The paper then presents co-training to encourage inter-frame consistency and resampling to alleviate the non-zero-SNR problem in multi-round generation.Source: §3.1 · p. 2
  • The overview presents temporal-module co-training for inter-frame consistency and resampling for longer multi-round generation.It also introduces FF-drive1, a dataset of detailed video-text pairs for long-video generation.Source: §3.1 · p. 2

3.2 Model Architecture

FlexiFilm uses a VAE, a latent-space 3D U-Net, and a temporal conditioner for multimodal references. Its video projector derives per-frame spatial features and then combines them temporally to guide generated frames.

3.2 Model Architecture

  • The model comprises a pretrained VAE, a pretrained VideoCrafter-1 3D U-Net, and a temporal conditioner that accepts long text descriptions, images, or frames as references.The VAE maps input video frames to latents, and diffusion operates on those latents.Source: §3.2 · p. 2, Figure 2 · p. 3
  • The temporal conditioner fuses visual references and text to guide generation with spatial and temporal information.The projected features are sent to 3D U-Net spatial transformers for frame-wise cross-attention with the noisy video latent.Source: §3.2 · pp. 2–3, Figure 2 · p. 3
  • The described pipeline produces generated frames using a VAE decoder, with a denoising video latent and Gaussian noise among its stated terms.The passage identifies the denoising latent as z-hat, epsilon as random Gaussian noise, and E as the text encoder.Source: §3.2 · p. 3
  • In the video projector, IP samplers extract independent spatial features from condition frames, then a temporal transformer models their relationships before projected features guide generated frames through frame-wise attention.For the described two-condition, four-generation example, the last condition-frame features are padded to four frames before temporal processing.Source: §3.2 · pp. 2–3

3.3 Training Strategy

FlexiFilm varies overlap between condition and target frames during training, then co-trains temporal modules in the conditioner and U-Net to model their relationship. A mixed conditional and unconditional training strategy determines which parameters are updated.

3.3.1 Scalable Overlapping Design.

  • Variable overlap between conditional and target frames is intended to make training easier and faster, with more overlap bringing generated-video distributions closer to the conditions.The design supports both image-to-video and frames-to-video generation.Source: §3.3.1 · p. 3

3.3.2 Temporal-Consist Co-Training.

  • Co-training the conditioner’s and U-Net’s temporal modules models the relationship between conditional frames and generated latents while other model parts remain frozen.The trainable parameters are in the temporal attention modules.Source: §3.3.2 · pp. 3–4
  • At p=0.1, the unconditional model is trained 10% of the time and the conditional model 90%; the former updates only the U-Net backbone, while the latter updates the conditioner and backbone.The authors report that this setting performs best, attributing it to adequate conditioner training and sufficient light fine-tuning of the pretrained U-Net.Source: §3.3.2 · p. 4

3.4 Inference Strategy

Multi-round inference extends video generation by conditioning each round on frames from the preceding round, but non-zero SNR can lead to overexposure and accumulating errors. FlexiFilm rescales latents before classifier-free guidance as a strategy for multi-round inference.

3.4.1 Multi-Round Inference for Longer Video Generation.

  • Recursive inference uses the last few frames from each f-frame video as the next round’s condition, theoretically generating m*f frames across m rounds.The authors note that non-zero SNR can cause overexposure under high classifier-free guidance and rapid collapse as rounds increase.Source: §3.4.1 · p. 4

3.4.2 Resampling Strategy for Non-Zero SNR Problem.

  • The non-zero SNR problem arises because the scheduler does not enforce zero SNR at the last timestep, although inference starts from pure Gaussian noise; high classifier-free guidance can then overexpose outputs.In multi-round inference, the resulting errors accumulate and frames can distort and collapse.Source: §3.4.2 · p. 4
  • The resampling strategy rescales video latents to their original standard deviation before classifier-free guidance, with scale r controlling the rescaling strength.The authors propose it to enable multi-round inference without harming video quality.Source: §3.4.2 · p. 4

3.5 Dataset Construction

FlexiFilm’s FF-Drive1 dataset was built from autonomous-driving videos to support long video generation in complex real-world scenes. The paper describes its construction and compares FlexiFilm’s long-video outputs with shorter baseline clips.

3.5.1 Insight for the Dataset.

  • FF-Drive1 uses autonomous-driving scenarios because long-video generation requires spatial and temporal consistency over time and training videos from complex real-world scenes.The dataset is intended to model the complexity of real driving scenarios.Source: §3.5.1 · p. 4
  • FlexiFilm generates clips of 200+ frames, while SVD and VideoCrafter are described as producing less consistent clips of 32- frames.The comparison also notes that SVD preserves building size as a vehicle moves, while VideoCrafter differs from the reference image and lacks inter-frame consistency.Source: Figure 5 · p. 5
  • The dataset combines autonomous-driving videos from eight open-source sources: BDD-Nexar, nuScenes, vkitti2, Argoverse 2, Brain4Car, ApolloScape, Kitti-MOT, and HOD.The passage lists the sources with bibliography markers [38], [6], [5], [35], [19],, [31], and [11], respectively.Source: §3.5.1 · p. 5

3.5.2 Construction Process.

  • Video LLaVA is queried through multiple dialogue rounds about scene and driving details, then used to summarize each video into its final caption.The captions are manually filtered for legal compliance, and driver faces are removed for privacy.Source: §3.5.2 · p. 5

4 EXPERIMENTS

FlexiFilm is evaluated qualitatively and quantitatively for long-video generation, visual quality, and inter-frame consistency. Tests include image-to-video comparisons and fine-tuning for natural-distribution videos.

4.1 Experiment Settings

  • The experiments compare FlexiFilm and baselines qualitatively with image and text conditions, then quantitatively on visual quality and structure-consistency indicators.The experiments are intended to assess long, consistent generation with flexible visual conditions and long text descriptions.Source: §4.1 · p. 5

4.2.1 Comparison with Baselines.

  • With one conditional frame, FlexiFilm generates clips exceeding 200 frames, while VideoCrafter-1 and SVD generate 32-frame clips lacking temporal consistency.The comparison uses one conditional frame because the baselines cannot take multiple frames.Source: §4.2.1 · pp. 5–6

4.2.2 Long and Short Video Generation in Different Domains.

  • Fine-tuning on Pexels-1k daily-life clips yields natural-distribution videos described as consistent between frames and faithful to conditions.The dataset contains 1,000 short video clips with paired descriptions.Source: §4.2.2 · p. 6, Figure 7 · p. 6

4.3.1 Visual Quality.

  • In zero-shot video generation on UCF101, FlexiFilm achieves better video quality in FVD scores than the baseline; its score was calculated at 256x256 resolution for 16 frames, conditioned on a class label and one image.Other model scores were excerpted from the corresponding papers, which used different FVD calculation settings.Source: §4.3.1 · p. 6

4.3.2 Consistency Indicator.

  • FlexiFilm outperforms VideoCrafter-1 in inter-frame consistency, measured as the average Clip Score of adjacent-frame pairs.Figure 6 presents the consistency-score comparison.Source: §4.3.2 · p. 6, Figure 6 · p. 6

4.3.3 Other Indicators.

  • For PSNR and SSIM, the authors randomly sample 1,000 training-set examples and report measurements for models trained with different methods.The passage identifies the evaluation setup but does not state the measurement outcomes.Source: §4.3.3 · p. 8

5 ABLATION STUDY

Ablations examine co-training choices, trainable temporal blocks, overlap-noise initialization, and resampling. The reported best settings and observed effects differ across these training and inference choices.

5.1 Ablation Study on Co-Training

  • An unconditional rate of 0.1 is the best setting among the six co-training models tested with different unconditional probabilities and conditional-frame counts.The authors identify unconditional rate, conditional frames, and trainable blocks as key training factors.Source: §5.1 · p. 8

5.2 Ablation Study on Temporal Blocks

  • Training U-Net temporal blocks together with video-projector temporal blocks gives the best visual quality in FVD, PSNR, and SSIM; the authors identify temporal-only training as best for video generation.The compared blocks also include video-projector spatial blocks, and training may use full or partial parameters.Source: §5.2 · p. 8

5.3 Ablation Study on Inference Strategy

  • Generation quality is sensitive to CFG scale, and adding noise to overlapping frames as their initial noise encourages structurally faithful results.The ablation also varies the number of overlapping frames.Source: §5.3 · p. 8

5.4 Ablation Study on Resampling Strategy

  • Resampling alleviates color overexposure and helps maintain consistency during multi-round inference.The experiments compare different resampling scales across multiple inference rounds.Source: §5.4 · p. 8

6 CONCLUSION

FlexiFilm is a latent video diffusion model designed for long, consistent video generation. Its temporal conditioner, co-training method, and resampling strategy address temporal inconsistency and overexposure; empirical validation reports superior qualitative and quantitative performance over existing methods.

6 CONCLUSION

  • FlexiFilm combines a temporal conditioner, co-training, and resampling, and demonstrates superior qualitative and quantitative performance over existing methods.The conclusion frames these components as addressing temporal inconsistency and overexposure in diffusion-based video generation.Source: §6 · p. 9

Related papers

Loading 2404.18620v1…