PaperlayerView exact revision on arXiv

Source-linked AI summary

Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising

Fu-Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, Hongsheng Li

arXiv:2305.18264v1cs.CV

TL;DR

Existing video diffusion methods are typically restricted to short videos and single text conditions, despite real videos containing multiple semantic segments. Gen-L-Video extends short-video diffusion models through temporal co-denoising to generate and edit consistent long videos without additional training. Experiments across mainstream text-to-video paradigms verify the framework’s broad applicability and scalability.

  • Problem

    Existing video generation and editing methods are generally limited to fewer than 24 frames and single text conditions, restricting support for long videos with changing semantic content.

  • Method

    Gen-L-Video treats long videos as overlapping short-video clips and uses temporal co-denoising to extend established short-video generation and editing methods without additional training.

  • Results

    Experiments across pretrained, tuning-free, and one-shot-tuning text-to-video methods verify that Gen-L-Video is universal and scalable for long-video generation and editing.

  • Takeaways & Limitations

    The framework broadens short-video diffusion models to multi-text conditioned long-video generation and editing while supporting additional controls such as layout control and video inpainting.

  • Takeaways & Limitations

    The framework has not been tested with the co-working of different video diffusion models with different lengths.

Abstract

from arXiv · show

Leveraging large-scale image-text datasets and advancements in diffusion models, text-driven generative models have made remarkable strides in the field of image generation and editing. This study explores the potential of extending the text-driven ability to the generation and editing of multi-text conditioned long videos. Current methodologies for video generation and editing, while innovative, are often confined to extremely short videos (typically less than 24 frames) and are limited to a single text condition. These constraints significantly limit their applications given that real-world videos usually consist of multiple segments, each bearing different semantic information. To address this challenge, we introduce a novel paradigm dubbed as Gen-L-Video, capable of extending off-the-shelf short video diffusion models for generating and editing videos comprising hundreds of frames with diverse semantic segments without introducing additional training, all while preserving content consistency. We have implemented three mainstream text-driven video generation and editing methodologies and extended them to accommodate longer videos imbued with a variety of semantic segments with our proposed paradigm. Our experimental outcomes reveal that our approach significantly broadens the generative and editing capabilities of video diffusion models, offering new possibilities for future research and applications. The code is available at https://github.com/G-U-N/Gen-L-Video.

1 Introduction

Existing text-driven video methods generally support fewer than 24 frames and a single text condition, whereas Gen-L-Video extends short-video diffusion models to consistent, multi-segment long-video generation and editing without additional training.

  • 1 Introduction: Current video generation and editing methods are usually limited to fewer than 24 frames and a single text condition.These constraints do not match videos whose content changes over time.
  • 1 Introduction: Gen-L-Video extends short-video diffusion models to long videos with multiple semantic segments without additional training.It treats long videos as collections of short clips while targeting consistent content across segments.
  • 1 Introduction: Naively splicing independently denoised short clips produces noticeable content and detail discrepancies between clips.The paper illustrates this issue with drastic color changes in a jeep across video clips.
  • 1 Introduction: The framework was evaluated with pretrained, tuning-free, and one-shot-tuning text-to-video methods, all yielding favorable results.It also supports additional controls and applications including layout control and arbitrary-object video inpainting.

2 Related Work

Related work spans text-to-image diffusion, pretrained, tuning-free, and one-shot-tuning text-to-video methods, alongside autoregressive and diffusion approaches to long-video generation. Gen-L-Video builds on this landscape by applying temporal co-denoising to overlapping short-video windows.

  • 2 Related Work: Long-video generation has used autoregressive or diffusion models, while Gen-L-Video instead applies temporal co-denoising over overlapping short-video clips.The framework’s overview describes overlapping windows as the basis for extending short-video models to arbitrary-length videos.
  • 2 Related Work: Text-to-image research progressed from GANs and autoregressive transformers to diffusion models with controllable layouts and editing capabilities.Representative developments include classifier-free guidance, latent diffusion, ControlNet, and T2I-Adapter.
  • 2 Related Work: Pretrained text-to-video methods add temporal modeling to text-to-image or video diffusion architectures, but large-scale text-video data and temporal consistency remain challenges.Examples include GODIVA, NÜWA, CogVideo, Video Diffusion Models, Imagen Video, Make-A-Video, MagicVideo, and LVDM.
  • 2 Related Work: Tuning-free text-to-video methods adapt pretrained text-to-image models for video while adding mechanisms to maintain consistency across frames.Frame-wise image editing alone can create inconsistencies because of limited temporal awareness.
  • 2 Related Work: One-shot-tuning methods fine-tune a pretrained text-to-image model on one video to reproduce similar motion or content, offering editing flexibility at additional training cost.Tune-A-Video and SinFusion represent this direction, while earlier single-video GANs incur substantial computational burden.

3 Method

Gen-L-Video performs parallel temporal co-denoising over overlapping short video clips, combining their outputs to extend off-the-shelf short-video diffusion models to long, multi-text videos without additional training.

  • Temporal Co-Denoising: The method denoises overlapping short clips in parallel and approximates the long-video denoising trajectory through their joint temporal process.Each clip is projected from the full video trajectory, denoised independently, and combined to recover the long-video transition.
  • Diffusion Preliminaries: The underlying diffusion process transforms noisy video toward clean video using conditional noise prediction and classifier-free guidance.The model progressively denoises from vT sampled from Gaussian noise to v0, with c represented by one or multiple text prompts.
  • Temporal Co-Denoising: Short clips use fixed length M and stride S, with S=M//2 or M//4 empirically providing strong results while S=M reduces the method to isolated denoising.The total video length is S*N + M, and overlapping clips share frames for joint denoising.
  • Temporal Co-Denoising: 34?
  • Temporal Co-Denoising: Gen-L-Video extends established short-video generation and editing methods to videos of any length without additional training.The framework constructs an abstract long-video generator and editor from off-the-shelf short-video diffusion models.
  • Condition Interpolation: Sparse text conditions are propagated to unlabeled clips through adjacent interpolation, enabling smoother guidance with fewer user-provided prompts.The interpolated conditions are derived between neighboring labeled clips.

4 Integrate Gen-L-Video with Mainstream Paradigms

Gen-L-Video integrates temporal co-denoising with pretrained, tuning-free, and one-shot-tuning video-generation paradigms for long-video generation and editing. The framework also supports personalized layout control and arbitrary-object editing through additional models.

  • Mainstream paradigms: Gen-L-Video is applied to pretrained, tuning-free, and one-shot-tuning Text-to-Video paradigms for long-video generation and editing.The implementations build on pretrained LDM-based models and adapt each paradigm to the proposed framework.
  • Temporal co-denoising: Bidirectional cross-frame attention replaces sparse causal attention to improve consistency across long videos.The method is implemented in tuning-free and one-shot-tuning pipelines, while temporal co-denoising provides the long-video generation pipeline.
  • One-shot tuning: Clip identifiers guide one-shot-tuning denoising for each short video clip, while randomly dropping them during training reduces content overfitting.The clip identifier is introduced to associate each clip with its denoising process, but can otherwise weaken text-based editing.
  • Controllable generation: Personalized diffusion models and pose or segmentation controls extend the framework to controllable video generation and editing.The framework combines fine-tuned models such as DreamBooth or LoRA with ControlNet and T2I-Adapter controls.
  • Arbitrary-object editing: Open-set detection and segmentation enable precise arbitrary-object inpainting while preserving other video content.Grounding DINO detects the target object and SAM produces frame-level masks for inpainting.

5 Experiments

Experiments evaluate Gen-L-Video on varied long videos using qualitative, quantitative, and ablation studies. Results cover multiple generation and editing settings, while comparisons assess frame consistency and textual alignment against isolated denoising.

  • Benchmark: The benchmark contains 66 videos ranging from 32 to hundreds of frames, with source prompts and four edited prompts per video.Edited prompts cover object changes, background changes, style transfer, similar-motion changes, and multiple changes.
  • Qualitative results: Gen-L-Video produces favorable qualitative results across varied lengths, paradigms, multi-text conditions, layout controls, personalization, and video inpainting.The qualitative results include pretrained, tuning-free, and one-shot-tuning Text-to-Video systems as well as controlled and edited videos.
  • Quantitative results: Quantitative evaluation compares Gen-L-Video with Isolated Denoising using frame consistency and textual alignment metrics plus human preference votes.Frame consistency uses average pairwise cosine similarity between frame embeddings; textual alignment uses CLIP-score averages and variance.
  • Ablation study: Bidirectional cross-frame attention yields smoother, more consistent results than sparse causal attention in the ablation study.Sparse causal attention typically makes the first frames inconsistent with later frames.
  • Ablation study: Removing the clip identifier causes one-shot-tuning generation to fail on consecutive source-video content under random initial noise.The corresponding method with the clip identifier succeeds in the comparison.

6 Conclusion

The paper presents Gen-L-Video as a universal and scalable way to extend short video diffusion models to efficient multi-text conditioned long-video generation and editing. It also identifies untested cooperation among different video diffusion models and lengths as future work.

  • Conclusion: Gen-L-Video extends short video diffusion models to efficient multi-text conditioned long-video generation and editing.The framework integrates mainstream Text-to-Video methods and adds improvements for long-video use.
  • Conclusion: The framework’s universality and scalability are supported by experiments across mainstream Text-to-Video paradigms.The conclusion characterizes the methodology as universal and scalable.
  • Limitations: The paper has not experimented with combining different video diffusion models with different lengths for greater generation and editing flexibility.This combination is identified as future work.

I Dataset Details

The evaluation dataset contains videos from the TGVE competition and the internet, with prompts designed to test several types of dynamic editing changes. Each video is paired with a source prompt and multiple edited prompts.

  • Prompt design: Internet videos receive four prompts covering object recognition, style, background, similar motion, or combinations of these changes.These prompts are designed relative to the original prompts.

II User Study Details

The user study compared Gen-L-Video with isolated denoising on frame consistency and textual alignment. Participants reported similar textual alignment but clearer alignment stability for Gen-L-Video.

  • Participants reported no significant textual-alignment difference between Gen-L-Video and isolated denoising.The study generated videos from varying prompts and asked participants which video better matched the text.
  • Gen-L-Video showed a clear improvement in alignment stability while maintaining good text-based editing capabilities.
  • The comparison evaluated frame consistency and textual alignment using participant judgments across shuffled video pairs.The study used 1,040 shuffled videos for frame-consistency judgments and 279 videos for textual-alignment judgments.

III Proof for the Optimal Approximation

The proof formulates optimal long-video approximation as a weighted quadratic objective over corresponding frames in overlapping short clips. Setting its gradient to zero yields each long-video frame as a weighted combination of corresponding clip frames.

  • The optimal long-video approximation is obtained by solving a quadratic optimization problem that makes short-clip reconstructions close to the target frame.The objective includes pixel-wise weights for the video clips and treats terms independent of the optimized frame as a constant.
  • The proof groups short clips containing a given long-video frame and rewrites the objective over those corresponding frames.For each target frame, the construction defines the relevant clip set and corresponding frame indices before optimizing.
  • After differentiating the objective with respect to the target frame and setting the gradient to zero, the weighted-average solution follows.
  • Each long-video frame equals the weighted sum of corresponding frames from all short clips that contain it.The weights are pixel-wise, and the tensor product combines each clip frame with its weighting mask.

IV Additional Results

Additional experiments show Gen-L-Video extending short-video diffusion models to multi-text, controllable, editable, and semantically evolving long videos. The examples emphasize improved cross-clip relevance, precise layout control, consistent inpainting, and smooth semantic changes.

  • The additional results include long-video generation, controllable generation, video editing, and smooth semantic transitions.
  • Multi-text long video: For multi-text conditioning, the video is split into short clips with distinct content changes, lengths, strides, and text prompts.Different colors indicate clips associated with different text prompts.
  • Pretrained Text-to-Video: Gen-L-Video significantly enhances relevance between clips when extending pretrained short-video models to longer video generation.The comparison is against isolated denoising in Fig. 7.
  • Controllable video generation: The method combines with additional control information to achieve precise layout control in long-video generation.These controllable-generation results are shown in Fig. 8.
  • Edit anything: Gen-L-Video reliably edits very long videos while maintaining consistent content, including examples longer than 12 and 20 seconds.The inpainting examples are presented in Figs. 9 and 10.
  • Long video with smooth semantic changes: The paradigm supports smooth semantic changes, such as transforming a road scene from day to night to represent passing time.The generated results are shown in Fig. 11.
Loading 2305.18264v1…