FIFO-Diffusion: Generating Infinite Videos from Text without Training

1 Introduction

• Diagonal denoising processes frames at increasing noise levels, allowing noisier frames to reference cleaner preceding frames but creating a training-inference gap.Latent partitioning reduces this gap, while lookahead denoising exploits forward references to less noisy frames.Source: §1 · p. 2
• FIFO-Diffusion generates arbitrarily long videos without additional training from video diffusion models pretrained on short clips.The method uses a queue-based diagonal denoising process that outputs fully denoised frames while adding new noisy frames.Source: §1 · p. 2
• FIFO-Diffusion uses constant memory regardless of generated video length and supports straightforward parallel inference across multiple GPUs.Source: §1 · p. 2
• Experiments on four U-Net- and DiT-based baselines show extremely long videos with natural motion and no quality degradation over time.Source: §1 · p. 2

2 Text-to-Video Diffusion Models

• Text-to-video diffusion models use an encoder, decoder, and noise-prediction network to learn text-conditioned video distributions.Videos are represented as sequences of latent frame representations before diffusion denoising.Source: §2 · p. 2
• A video latent is formed by concatenating encoded frames, then perturbed with Gaussian noise at a sampled diffusion timestep.The perturbed latent uses predefined scale and noise coefficients, with the text condition supplied to the model.Source: §2 · pp. 2–3
• Figure 2 illustrates diagonal denoising, where frames with increasing noise levels are processed together and the fully denoised frame is dequeued while new random noise is enqueued.Source: Figure 2 · p. 3
• The model generates video by iteratively denoising an initial random latent over a scheduled sequence of diffusion steps using a sampler such as DDIM.Source: §2 · p. 3

3.1 Diagonal denoising

• FIFO-Diffusion generates arbitrary-length videos by diagonal denoising a queue, dequeuing each fully denoised frame and enqueuing a new noisy latent.The method initializes the queue from random noise and uses a first-in-first-out process without additional training.Source: §3.1 · p. 3
• The sliding-window procedure uses O(f) memory per step for N frames, independent of N, and propagates context with stride 1 to later frames.Unlike chunked autoregressive methods that condition mainly on the last generated frame, FIFO-Diffusion allows each frame to reference sufficient preceding context.Source: §3.1 · p. 4

3.2 Latent partitioning

• Latent partitioning extends the queue to nf frames, divides it into n blocks, and processes each block independently to narrow the maximum noise-level difference.Under boundedness and Lipschitz assumptions, the diagonal-denoising error is bounded by the noise-level difference, so narrowing that difference reduces the gap.Source: §3.2 · pp. 4–5
• Latent partitioning also supports parallel multi-GPU inference and uses nf inference steps, reducing discretization error while increasing inference steps with the extended queue.Its throughput benefit comes from processing partitioned blocks in parallel.Source: §3.2 · p. 4

3.3 Lookahead denoising

• Lookahead denoising updates only the later half of the queue with stride f′ = ⌊f/2⌋, ensuring each updated frame references at least f′ clearer frames.The method requires twice the computation of original diagonal denoising, but this overhead can be addressed through parallelization; Figure 5 reports improved noise-prediction performance except at a few early time steps.Source: §3.3 · pp. 5–6, Figure 5 · p. 6

4.2 Qualitative results

• FIFO-Diffusion generates videos longer than 10K frames from a 16-frame pretrained baseline while preserving visual quality and semantic information across later frames.The method also maintains motion consistency and supports extensive motion driven by changing prompts.Source: §4.2 · p. 6
• FIFO-Diffusion significantly outperforms FreeNoise, Gen-L-Video, and LaVie + SEINE in motion smoothness, frame quality, and scene diversity.FreeNoise struggles with dynamic scenes, Gen-L-Video often blurs backgrounds, and LaVie + SEINE accumulates errors and shows chunk discontinuities.Source: §4.2 · p. 8

4.3 Quantitative results

• FIFO-Diffusion outperforms all compared long-video methods on FVD128 and IS, including PVDM-L (400-400s), while using 64 inference steps without additional training.PVDM-L iteratively generates 16 frames over 400 diffusion steps, whereas FIFO-Diffusion uses lookahead denoising with 64 steps.Source: §4.3 · p. 8, Table 1 · p. 8

4.4 Computational cost

• FIFO-Diffusion uses constant memory for arbitrary video lengths, while FreeNoise scales memory with target length and Gen-L-Video is slowest because of redundant computation.Multiple GPUs can reduce FIFO-Diffusion sampling time, although lookahead denoising increases computational demand.Source: Table 2 · p. 8, §4.4 · p. 9

4.5 Ablation study

• Latent partitioning improves visual quality, temporal consistency, and training-inference alignment, while lookahead denoising further reduces flickering and relative MSE.Together, the methods achieve relative MSE losses below 1.0 when lookahead denoising is combined with latent partitioning.Source: §4.5 · p. 9, Table 3 · p. 9

5.2 Long video generation

• Long-video methods generally predict future frames autoregressively or generate frame sets hierarchically, with approaches including Video LDM, MCVD, FDM, and SEINE.Autoregressive methods can produce indefinitely long videos but depend on sequential prediction, while masked-learning methods support prediction and interpolation.Source: §5.2 · p. 10

5.3 Diffusion models with latents of different noise levels

• Sliding-window diffusion methods use temporally varying noise levels and train diffusion models from scratch, incurring substantial computational costs for text-to-video generation.FIFO-Diffusion instead performs training-free inference with standard models trained on uniformly noisy latents.Source: §5.3 · p. 10
• FIFO-Diffusion combines frame index and diffusion time into a one-dimensional queue, replacing the nested-loop processing used by a prior sequence-generation method.This design improves efficiency by using a single loop.Source: §5.3 · p. 10

• FIFO-Diffusion enables infinitely long video generation from text without tuning video diffusion models pretrained on short clips.Source: §6 · p. 10
• Diagonal denoising processes increasingly noisy latents in a first-in-first-out queue, while latent partitioning reduces the training-inference gap and lookahead denoising exploits cleaner reference frames.Source: §6 · p. 10
• The training-inference gap persists because diagonal denoising changes the model’s input distribution, motivating future integration of the paradigm into training.Source: §6 · p. 10

• The proof begins from bounded noise prediction, asserting a finite bound M for ||ϵ(·)||.Source: §A.1 · p. 13
• During inference, latents and noise levels lie in bounded ranges with σmin > 0, supporting analysis of the predictor over the relevant domain.Source: §A.2 · p. 13
• Gaussian-smoothed data terms are twice differentiable and continuous, while their denominator is bounded below, making the differential function bounded and Lipschitz continuous.Source: §A.2 · p. 13
• Because ϵθ estimates the ideal predictor ϵ, the appendix treats Lipschitz continuity of ϵθ as a justified assumption.Source: §A.2 · p. 13

Implementation details

• Experiments use VideoCrafter1, VideoCrafter2, zeroscope, Open-Sora Plan, LaVie, and SEINE as pretrained models, with randomly generated prompts for non-automated results.Source: B Implementation details · p. 14
• Latent partitioning uses n = 4 partitions, while the provided algorithms cover diagonal denoising, initial latent construction, latent partitioning, and lookahead denoising.Source: B Implementation details · p. 14, C Algorithms of FIFO-Diffusion · pp. 15–16

Evaluation

• The experiments randomly generate 30 prompts and 100-frame videos, then compare FIFO-Diffusion with FreeNoise across preference, motion, text fidelity, and aesthetic criteria.Source: §B.1 · p. 14

Algorithms

• The FIFO-Diffusion procedures update queued latent segments over scheduled timesteps, and the pseudocode marks key denoising loops as parallelizable.Source: C Algorithms of FIFO-Diffusion · pp. 15–16

Qualitative results

• Additional qualitative results use VideoCrafter2, VideoCrafter1, zeroscope, and Open-Sora Plan, with figures documenting generated videos and frame indices.Source: D Qualitative results of FIFO-Diffusion · p. 17, Figure 10 · p. 17, Figure 11 · p. 18

Multi-prompt generation

• Multiple prompts are applied sequentially by changing the conditioning prompt over specified frame intervals.Source: §E.1 · p. 23

Qualitative results

• Figures 16 and 17 provide qualitative multi-prompt generation results based on VideoCrafter2.Source: §E.2 · p. 23
• The displayed videos use three prompts in Figure 16 and two prompts in Figure 17, with each frame labeled by its index.Source: Figure 16 · p. 23, Figure 17 · p. 24

• Figures 18 and 19 compare FIFO-Diffusion qualitatively with Gen-L-Video, FreeNoise, and LaVie + SEINE.Source: F Qualitative comparisons with other… · p. 25, Figure 18 · p. 25, Figure 19 · p. 26
• The comparison covers multiple long-video generation techniques rather than a single baseline.Source: F Qualitative comparisons with other… · p. 25
• The compared examples use frame indices in the top-left corner of each frame to indicate temporal position.Source: Figure 18 · p. 25, Figure 19 · p. 26

Motion evaluation

• Over 65% of FreeNoise videos fall in the first optical-flow bin, whereas FIFO-Diffusion produces a broader motion range.The videos use randomly sampled prompts from the MSR-VTT test set.Source: G Motion evaluation · p. 27, Figure 20 · p. 27

Ablation study

• The ablation compares diagonal denoising alone, diagonal denoising plus latent partitioning with n=4, and the full addition of lookahead denoising.DD, LP, and LD denote diagonal denoising, latent partitioning, and lookahead denoising, respectively.Source: H Ablation study · p. 28, Figure 21 · p. 28, Figure 22 · p. 29

Potential Broader Impact

• The paper notes that pretrained video diffusion models can potentially synthesize unexpectedly inappropriate content because the method requires no additional training.The authors also state that the method could mildly address ethical concerns associated with generative-model training data.Source: I Potential Broader Impact · p. 29