PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text

Roberto Henschel, Levon Khachatryan, Hayk Poghosyan, Daniil Hayrapetyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, Humphrey Shi

arXiv:2403.14773v2cs.CVcs.AIcs.CLcs.LGcs.MMeess.IV

TL;DR

Existing video generators mainly target short sequences, while naive autoregressive extension produces inconsistent transitions, stagnation, or quality degradation. StreamingT2V combines short- and long-term memory with randomized blending to generate and enhance extended videos. Experiments report temporally consistent long videos with motion and without content stagnation.

  • Problem

    Existing approaches mostly generate short videos, and naive autoregressive extension can produce inconsistent transitions, appearance changes, stagnation, or quality degradation in long videos.

  • Method

    StreamingT2V autoregressively generates long videos using CAM for preceding-chunk conditioning, APM for preserving scene and object features, and randomized blending for chunk enhancement.

  • Results

    StreamingT2V generates long, temporally consistent videos from text without video or content stagnation, while competing image-to-video methods exhibit stagnation or degradation.

  • Takeaways & Limitations

    The approach provides a seamless text-to-long-video generator with consistency and motion, and its framework generalizes to DiT architectures such as OpenSora.

Abstract

from arXiv · show

Text-to-video diffusion models enable the generation of high-quality videos that follow text instructions, making it easy to create diverse and individual content. However, existing approaches mostly focus on high-quality short video generation (typically 16 or 24 frames), ending up with hard-cuts when naively extended to the case of long video synthesis. To overcome these limitations, we introduce StreamingT2V, an autoregressive approach for long video generation of 80, 240, 600, 1200 or more frames with smooth transitions. The key components are:(i) a short-term memory block called conditional attention module (CAM), which conditions the current generation on the features extracted from the previous chunk via an attentional mechanism, leading to consistent chunk transitions, (ii) a long-term memory block called appearance preservation module, which extracts high-level scene and object features from the first video chunk to prevent the model from forgetting the initial scene, and (iii) a randomized blending approach that enables to apply a video enhancer autoregressively for infinitely long videos without inconsistencies between chunks. Experiments show that StreamingT2V generates high motion amount. In contrast, all competing image-to-video methods are prone to video stagnation when applied naively in an autoregressive manner. Thus, we propose with StreamingT2V a high-quality seamless text-to-long video generator that outperforms competitors with consistency and motion. Our code will be available at: https://github.com/Picsart-AI-Research/StreamingT2V

1. Introduction

Existing text-to-video methods generally generate short sequences, making long-video applications difficult because naive autoregressive extension causes inconsistent transitions and stagnation. StreamingT2V addresses this with short- and long-term memory modules plus randomized blending for seamless extended generation.

  • Most existing video generators focus on short sequences of 16, 24, or 384 frames, limiting applications such as advertising and storytelling.
  • Training video generators directly on long videos of at least 1200 frames is usually impractical because even short-sequence training is expensive.Generating 16 frames can require 260K steps and a 4.5K batch size.
  • Autoregressively concatenating noisy latents or CLIP embeddings from preceding chunks provides weak conditioning and leaves scene transitions inconsistent.
  • All assessed image-to-video methods produce video stagnation or strong quality degradation when autoregressively conditioned on preceding frames.
  • StreamingT2V uses CAM for short-term continuity and APM for preserving object and global-scene characteristics across autoregressive generation.CAM borrows preceding-frame content without restricting motion, while APM addresses appearance changes and quality decline over time.
  • A randomized blending approach enables autoregressive enhancement of overlapping chunks, while experiments report temporally consistent long videos without content stagnation.

2. Related Work

Related work extends short-video or image-generation systems toward longer videos, but reported approaches remain constrained by limited motion, scene cuts, quality degradation, or inconsistent chunk conditioning. These limitations motivate stronger mechanisms for coherent long-video generation.

  • Early text-guided video diffusion models generated low-resolution videos of up to 16 frames without autoregression and required substantial training resources.
  • Spatial and temporal upsampling can produce higher-resolution and longer videos, but content remains limited by the depicted key frames.
  • T2V0 and ART-V generate longer videos with text-to-image diffusion models, but their lack of global reasoning leads to simple, unnatural, or repetitive motion.
  • MTVG and FIFO-Diffusion convert text-to-video models into autoregressive systems, but strong consistency priors result in low motion and near-static content.
  • Anchor-based or linearly propagated conditioning methods can cause drastic scene cuts or severe quality degradation because chunk consistency is not enforced.
  • CLIP-based conditioning and image cross-attention improve quality or consistency in some cases but remain limited by quality degradation and frequent cross-chunk inconsistencies.
  • SparseCtrl requires additional black frames and a frame mask, creating an input inconsistency associated with frequent and severe scene cuts.

3. Preliminaries

StreamingT2V is built on latent-space video diffusion: an encoder maps video frames to latent codes, noise is added through a forward process, and a learned noise predictor supports text-conditioned denoising. Its pipeline initializes a short chunk, autoregressively generates subsequent content, and refines the long video.

  • The model operates in the latent space of a VQ-GAN autoencoder, encoding each video frame into a latent tensor and decoding generated latents frame-wise.
  • The diffusion forward process gradually adds Gaussian noise to the initial signal until the terminal latent is approximately standard Gaussian noise.
  • The learned backward process denoises from standard Gaussian noise toward a valid signal, while the model predicts the Gaussian noise conditioned on the text prompt.
  • StreamingT2V initializes the first 16-frame chunk with a pretrained text-to-video model, autoregressively generates later frames, and refines long videos with randomized blending.
  • Latent sequences can be denoted by frame ranges, such as the sequence from frame r through frame s.

4. Method

StreamingT2V generates long videos autoregressively by extending a short-video diffusion model with short-term and long-term memory. A randomized blending procedure enables higher-resolution enhancement while smoothing overlapping chunk transitions.

  • Pipeline: The pipeline first synthesizes 256 × 256 long videos, then enhances them to 720 × 720 resolution.Initialization creates a 16-frame chunk; subsequent content is generated autoregressively.
  • Conditional Attention Module: CAM conditions each new chunk on preceding-chunk features through frame encoding and cross-attention into UNet skip connections.Its feature extractor follows Econd and shared Video-LDM UNet layers, while feature injection uses temporal multi-head attention.
  • Conditional Attention Module: CAM provides short-term continuity while avoiding the severe inconsistencies caused by zero-padded convolutional conditioning.The module conditions F output frames on Fcond preceding frames without requiring additional zero-valued frames.
  • Appearance Preservation Module: APM uses a fixed anchor frame to inject scene and object information into text cross-attentions during autoregression.This long-term memory is designed to preserve appearance and identity features across generated chunks.
  • Autoregressive Video Enhancement: Randomized blending merges overlapping enhanced chunks by sharing noise and randomly selecting a blend position before subsequent denoising.Chunks contain 24 frames with 8-frame overlaps, and probabilistic latent mixing reduces overlap inconsistencies.

5. Experiments

The experiments evaluate temporal consistency, text alignment, quality, and motion on 240-frame videos against open-source autoregressive baselines. StreamingT2V achieves strong consistency and alignment while avoiding the motion stagnation and degradation seen in competitors.

  • Metrics: The evaluation measures temporal consistency, text alignment, and per-frame quality on videos generated with 240 frames.Metrics include SCuts, MAWE, and CLIP similarity.
  • Quantitative Results: StreamingT2V performs best in MAWE and CLIP score, while ranking second in SCuts because FreeNoise generates near-constant videos.MAWE combines motion amount and warp error; SCuts counts detected scene cuts.
  • Benchmark: The benchmark contains 50 prompts spanning different actions, objects, and scenes, with comparisons against image-to-video and video-to-video methods.All methods use released weights and hyperparameters, while the first 16-frame chunk and refiner are standardized where applicable.
  • Comparison with Baselines: OpenSoraPlan produces a 6 times higher SCuts score and SparseControl 100 times more scene cuts than StreamingT2V.The authors attribute SparseControl’s severe cuts to zero-padding conditional frames and inconsistent inputs.
  • Comparison with Baselines: StreamingT2V obtains the highest CLIP score among evaluated methods, while competing CLIP-conditioned methods are prone to misalignment.The authors hypothesize that applying a natural-image CLIP encoder to generated images introduces domain shift.
  • Comparison with Baselines: StreamingT2V maintains relatively stable MAWE and CLIP scores from 120 to 220 frames, whereas competitors show stagnation, degradation, or inconsistencies.Qualitative comparisons report frozen backgrounds and cameras for competitors, while StreamingT2V retains smooth motion and transitions.

6. Conclusion and Future Work

The paper concludes that long-video autoregression suffers from temporal inconsistency or stagnation, and presents StreamingT2V as a way to preserve scene features while maintaining motion. The framework also generalizes to OpenSora, though detailed analysis remains future work.

  • Conclusion: Existing methods generate long videos with temporal inconsistencies or severe stagnation up to standstill.StreamingT2V addresses this through short- and long-term dependencies for smooth continuation with high motion.
  • Conclusion: StreamingT2V combines short- and long-term dependencies with randomized blending to generate long text-conditioned videos without content stagnation.The approach preserves scene features while allowing a video enhancer inside the autoregressive process.
  • Future Work: The framework generalizes to DiT architectures, demonstrated by adding CAM and APM to OpenSora.Visual inspection confirmed this generalization, while detailed analysis is left for future research.

Appendix

The appendix extends the paper with additional experiments, qualitative results, ablations, implementation details, prompts, and the MAWE definition. It also reports a user study comparing motion, text alignment, and overall quality.

  • Overview: The appendix investigates text-to-video quality, longer sequences, implementation details, and experimental procedures.It complements the main paper with further analyses and results.
  • User Study: A user study compares StreamingT2V with prior work on motion, text alignment, and overall quality using pairwise video judgments.Videos are resized and cropped to reduce potential biases.
  • Additional Analyses: Additional sections provide qualitative long-video results, comparisons with competitors, and ablations of CAM, APM, and randomized blending.The appendix also includes implementation and training details.
  • Evaluation Details: The appendix supplies the benchmark prompts and the exact definition of the motion aware warp error metric.These materials support reproducibility and interpretation of the evaluation.

7. User Study

A user study compares StreamingT2V directly with competing methods on motion, text alignment, and overall quality. StreamingT2V is preferred more often across comparisons, including a 65% versus 17% motion preference against SEINE.

  • The user study uses one-to-one comparisons between StreamingT2V and competing methods, evaluating motion, text alignment, and overall quality.Participants answer one question for each criterion and choose the preferred model, equal results, or the competing model.
  • 65% of votes preferred StreamingT2V for motion quality, compared with 17% preferring SEINE.The passage attributes this preference to non-stagnating videos, temporal consistency, and seamless chunk transitions.
  • StreamingT2V was significantly more often preferred than competing methods across comparisons, including text alignment and overall quality.The reported preference is associated with competitors’ quality degradation over time.

8. Qualitative Results

Qualitative evaluations show that StreamingT2V maintains motion, scene and object details, temporal consistency, and smooth chunk transitions in long videos. Competing methods instead exhibit stagnation, quality degradation, object-detail loss, or inconsistent transitions.

  • Very Long Video Generation: StreamingT2V generates 1200-frame videos for varied actions and characters while preserving scene and object features across generation.The examples include dancing, running, camera movement, bees, and jellyfish; the passage credits APM with preserving appearance.
  • Qualitative Comparisons: StreamingT2V produces non-stagnating videos with good motion quality, seamless chunk transitions, and temporal consistency.These properties are reported across visual comparisons with competing methods.
  • Qualitative Comparisons: DynamiCrafter-XL and SVD show image-quality degradation over time, including incorrect colors, deteriorated backgrounds, and weaker textual alignment.The examples include wrong colors on a beagle’s face and deteriorating background patterns.
  • Qualitative Comparisons: SparseControl, FreeNoise, and SEINE produce near-static results or fail to follow camera actions such as zooming out.The reported failures prevent these methods from performing prompted actions in the examples.
  • Qualitative Comparisons: OpenSora produces little or no motion, while OpenSoraPlan loses initial object details and suffers substantial autoregressive quality degradation.OpenSora results include static videos or image warping without motion; in one example, the dog becomes hardly recognizable.
  • Chunk Transitions: X-T slice visualizations show smooth StreamingT2V transitions, whereas CLIP conditioning or latent concatenation can create strong inconsistencies between chunks.The comparison directly examines temporal structure at chunk boundaries.

9. Ablation Studies

The ablations evaluate CAM, APM, and randomized blending, showing improvements in chunk consistency, identity and scene preservation, and enhancement smoothness.

  • 9.1. Conditional Attention Module: CAM achieves an SCuts score of 0.03, versus 0.24 for Conc-Cond and 0.284 for Add-Cond.This corresponds to an 88% lower SCuts score than the baselines.
  • 9.2. Appearance Preservation Module: APM preserves object identity and scene features across autoregressive generations, while removing long-term memory causes those features to change over time.The study specifically reports consistent facial details and scene characteristics with APM.
  • 9.2. Appearance Preservation Module: 93.42 versus 94.95 re-identification scores show improved identity preservation with APM.The values compare Ours w/o APM and Ours, respectively.
  • 9.2. Appearance Preservation Module: 0.192 versus 0.151 LPIPS scores show improved scene preservation with APM, exceeding 20% improvement.The values compare Ours w/o APM and Ours, respectively.
  • 9.3. Video enhancer improvements: Randomized blending produces smoother enhancement transitions than independent enhancement or shared noise baselines.Optical-flow standard deviations are 8.72, 6.01, and 3.32 for B, B+S, and StreamingT2V, respectively.

10. Implementation detail

The implementation trains CAM and APM around a pretrained Video-LDM and applies classifier-free guidance from text and an anchor frame during StreamingT2V generation.

  • Training: CAM training freezes pretrained Video-LDM weights and trains new CAM layers for 400K steps with batch size 8 and learning rate 5 · 10^-5.CAM is trained before CAM+APM training.
  • APM training: APM training randomly samples an anchor frame from the first 16 frames to align training with inference and provide high-level semantic information.Conditioning uses frames 17–24, while denoising uses frames 17–32.
  • Streaming T2V stage: StreamingT2V uses classifier-free guidance from both text and an anchor frame.The guided noise prediction combines text and anchor-frame conditioning before denoising.
  • Streaming T2V stage: Text and anchor guidance weights are both set to 7.5 in the experiments.Training independently drops text, anchor, or both conditions with 5% likelihood.
  • Architecture and enhancement: The image encoder used in CAM consists of stacked 2D convolutions, layer normalization, and SiLU activations.The video enhancer diffuses an input video using 600 steps.

11. Test set prompts

The appendix lists test prompts, enhancement settings, and MAWE’s motion-quality definition used to evaluate long-video generation.

  • 11. Test set prompts: The test set includes prompts involving animals, people, weather, landscapes, camera movement, time-lapse scenes, and abstract processes.Examples include dancing Santa Claus, a baby crying during camera zoom-out, and fluids changing colors.
  • 11. Test set prompts: Figure 15 compares randomized blending with two video-enhancer baselines for chunk-transition consistency.The baselines independently enhance chunks or use shared noise for consecutive chunks.
  • 11. Test set prompts: Figure 16 illustrates the Add-Cond baseline used in the CAM ablation.The figure is referenced as an implementation comparison for Sec. 9.1.
  • 11. Test set prompts: Table 12 reports hyperparameters for the Streaming T2V stage, with additional architectural settings attributed to the Modelscope report.The table is presented as supplementary implementation information.
  • 12. MAWE Definition: MAWE combines mean warp error with optical-flow score to characterize motion amount and temporal quality.OFS averages squared optical-flow magnitudes, while mean warp error measures warped-frame pixel differences excluding occlusions.
  • 12. MAWE Definition: MAWE is defined as W(V) divided by OFS(V).The paper reports that this measure is well-aligned with human perception.
Loading 2403.14773v2…