PaperlayerView exact revision on arXiv

Source-linked AI summary

From Slow Bidirectional to Fast Autoregressive Video Diffusion Models

Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Fredo Durand, Eli Shechtman, Xun Huang

arXiv:2412.07772v4cs.CV

TL;DR

Bidirectional video diffusion is costly and poorly suited to interactive, streaming generation because each frame depends on future frames. CausVid adapts a pretrained bidirectional diffusion transformer into a causal autoregressive model and applies video distribution matching distillation with asymmetric teacher supervision. It achieves competitive or superior reported quality while enabling faster streaming generation and applications beyond text-to-video.

  • Problem

    Bidirectional attention requires processing future frames to generate the current frame, causing latency and limiting interactive or streaming video generation.

    Source: §1 · p. 2

  • Method

    CausVid combines a causal autoregressive diffusion transformer with video distribution matching distillation from a bidirectional teacher and KV-cached inference.

    Source: §1 · pp. 2–3, §3.2 · p. 6, §4.1 · p. 7

  • Results

    CausVid achieves video quality on par with state-of-the-art bidirectional diffusion models while offering enhanced interactivity and speed.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The causal design supports streaming generation and applications including zero-shot image-to-video, video-to-video translation, and dynamic prompting.

    Source: §1 · p. 3

    Quality degrades for extremely long videos, latency remains constrained by the VAE design, and DMD reduces output diversity.

    Source: §6 · p. 11

Abstract

from arXiv · show

Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies. The generation of a single frame requires the model to process the entire sequence, including the future. We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly. To further reduce latency, we extend distribution matching distillation (DMD) to videos, distilling 50-step diffusion model into a 4-step generator. To enable stable and high-quality distillation, we introduce a student initialization scheme based on teacher's ODE trajectories, as well as an asymmetric distillation strategy that supervises a causal student model with a bidirectional teacher. This approach effectively mitigates error accumulation in autoregressive generation, allowing long-duration video synthesis despite training on short clips. Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models. It enables fast streaming generation of high-quality videos at 9.4 FPS on a single GPU thanks to KV caching. Our approach also enables streaming video-to-video translation, image-to-video, and dynamic prompting in a zero-shot manner.

1. Introduction

Existing video diffusion models deliver high quality but bidirectional dependencies create latency, quadratic scaling costs, and barriers to interactive streaming generation. CausVid adapts diffusion transformers to causal autoregressive generation and combines this design with distillation and KV caching for faster, interactive video synthesis.

Limitations of bidirectional diffusion

  • Bidirectional attention makes generating one frame require processing the entire sequence, including future frames unavailable during interactive use.This creates long latency and prevents continual generation conditioned on changing user inputs.Source: §1 · p. 2
  • Existing video diffusion models become increasingly slow and expensive as frame count grows because compute and memory costs scale quadratically.Many denoising steps further limit practical long-video generation.Source: §1 · p. 2

Autoregressive alternative

  • Autoregressive generation produces frames sequentially, allowing playback to begin after the first frame and supporting interactive control and longer videos.Its main challenge is error accumulation, as later frames build on potentially flawed earlier predictions.Source: §1 · p. 2

CausVid

  • CausVid introduces an autoregressive diffusion transformer with causal frame dependencies for fast and interactive video generation.The model is presented as an adaptation of diffusion-transformer video generation toward causal inference.Source: §1 · p. 2
  • CausVid uses supervision from all input frames and KV caching for sample-efficient training and efficient autoregressive inference.These mechanisms address training efficiency and inference latency within the causal architecture.Source: §1 · p. 3

2. Related Work

Prior autoregressive and diffusion-distillation approaches face computational or causal limitations, while CausVid combines causal generation with video diffusion distillation. The resulting design supports streaming generation, longer videos than training clips, and applications beyond text-to-video.

Autoregressive video generation

  • Token-based autoregressive video models are computationally expensive because each frame typically contains thousands of tokens.Earlier approaches also used regression or GAN losses for frame prediction.Source: §2 · p. 3
  • CausVid adapts autoregressive video generation through diffusion distillation, improving efficiency and making it competitive with bidirectional diffusion.The related-work discussion identifies this adaptation as the method’s distinction from prior approaches.Source: §2 · p. 3

Applications

  • The method extends beyond text-to-video to zero-shot image-to-video and interactive applications, including dynamic prompting and user-responsive generation.These capabilities are illustrated through the paper’s application figures and described as supported uses.Source: Figure 4 · p. 5

Video diffusion distillation

  • Most prior video distillation methods target short, non-causal videos, whereas CausVid distills a non-causal teacher into a causal student for streaming generation.CausVid is trained on 10-second videos and can generate indefinitely through sliding-window inference.Source: §2 · p. 5

3. Background

Video diffusion models progressively denoise noisy samples, often in a latent space, while their training and sampling can be expressed through noise prediction or score functions. Distribution matching distillation trains efficient few-step generators to match the teacher-induced data distribution.

Video Diffusion Models

  • Diffusion models generate samples by progressively denoising Gaussian noise toward the data distribution.Training uses samples formed by adding random noise to data examples.Source: §3.1 · p. 6
  • The denoiser is typically trained to predict the added noise, although clean-data and v-prediction targets are alternatives.These prediction schemes are related to the diffusion score function.Source: §3.1 · p. 6
  • At inference, sampling starts from full Gaussian noise and repeatedly computes a cleaner sample from the current noisy sample.The paper uses the score function as a general representation of diffusion models across prediction parameterizations.Source: §3.1 · p. 6
  • Video diffusion commonly operates in a VAE-compressed latent space, reducing the dimensionality of spatial and temporal video data.Video denoisers can use architectures including U-Nets and Transformers.Source: §3.1 · p. 6

Distribution Matching Distillation

  • Distribution matching distillation trains a few-step student to match the smoothed data distribution of a slow, multi-step teacher.Its reverse-KL gradient is approximated by the difference between data-distribution and generator-distribution score functions.Source: §3.2 · p. 6
  • DMD freezes the data score while training the generator score online from generated outputs; DMD2 extends the framework to multi-step generation.DMD2 uses partially denoised intermediate samples instead of only random-noise inputs.Source: §3.2 · p. 6

4. Methods

CausVid uses an autoregressive diffusion transformer to generate latent video chunks sequentially while preserving local temporal consistency. Its training combines ODE-based student initialization with asymmetric distillation from a bidirectional teacher, and inference uses KV caching for efficient streaming.

4.1. Autoregressive Architecture

  • Tokens in the current frame attend only to current and previous frames, not future frames.The architecture therefore supports causal frame generation rather than requiring the entire video sequence.Source: Figure 5 · p. 7
  • The autoregressive diffusion transformer generates latent frames sequentially, using bidirectional attention within chunks and causal attention across chunks.This prevents current chunks from attending to future chunks while capturing local temporal dependencies.Source: §4.1 · pp. 6–7
  • The model extends DiT with block-wise causal attention masks while preserving its core structure and pretrained bidirectional weights.This design enables reuse of pretrained weights for faster convergence.Source: §4.1 · p. 7

4.2. Bidirectional →Causal Generator Distillation

  • The method distills a many-step bidirectional diffusion model into a 4-step causal generator using asymmetric distillation.The bidirectional teacher supervises a causally constrained student through a distribution matching distillation loss.Source: Figure 6 · p. 8, §4.2 · p. 8

4.3. Student Initialization

  • The student is initialized from teacher-generated ODE solution pairs before DMD training to stabilize optimization.The ODE trajectories are sampled with a pretrained bidirectional teacher, and selected timesteps are used for regression-based initialization.Source: §4.3 · p. 8

4.4. Streaming Inference

  • During inference, KV caching supports sequential frame generation and allows use of a fast bidirectional attention implementation.Block-wise causal attention is unnecessary at inference time when cached keys and values are used.Source: §4.4 · p. 8

5. Experiments

Experiments show that CausVid combines strong short- and long-video quality with substantially faster inference, while distillation and bidirectional-teacher supervision reduce autoregressive degradation. The method also supports streaming translation, image-to-video generation, and videos exceeding 10 minutes.

Text to Video Generation

  • 94.7 temporal quality, 64.4 frame quality, and 30.1 text alignment are achieved on short-video generation, outperforming all evaluated baselines.The compared methods include CogVideoX, OpenSORA, Pyramid Flow, and MovieGen.Source: §5.1 · p. 9
  • 84.27 VBench-Long total score places the method first among officially evaluated video generation models.Source: §5.1 · p. 9
  • The distilled model remains comparable to its bidirectional teacher in human preference ratings while providing orders-of-magnitude faster inference.The study used 29 prompts and three ratings per model pair and found the model consistently outperformed MovieGen, CogVideoX, and Pyramid Flow.Source: §5.1 · p. 9

Long Video Generation

  • The method maintains image quality over approximately 30 seconds and outperforms long-video baselines in temporal and frame-wise quality while remaining competitive in text alignment.Most autoregressive baselines show quality degradation over time, whereas the proposed method is reported to prevent error accumulation.Source: §5.1 · p. 10, Table 2 · p. 10, Figure 8 · p. 10

Efficiency

  • 160× lower latency and 16× higher throughput than similarly scaled CogVideoX demonstrate the efficiency gain of the proposed method.The comparison covers 10-second, 120-frame videos at 640 × 352 resolution and includes text encoding, diffusion, and VAE decoding.Source: §5.1 · p. 10, Table 3 · p. 10

Ablation Studies

  • Distillation with a bidirectional teacher and ODE-based student initialization outperforms causal-teacher and initial ODE-fitted variants.The final student improves frame-by-frame quality over the bidirectional teacher but performs worse on temporal flickering and output diversity.Source: §5.2 · p. 10

6. Discussion

The method retains high-quality video generation but has important boundaries in very long videos, output diversity, and current streaming speed. The paper identifies longer-video quality degradation, reduced diversity from DMD, and engineering limits around 10 FPS as remaining constraints.

Long-video generation

  • Videos longer than 30 seconds can show quality degradation, leaving extreme-duration generation as an open challenge.The authors identify error accumulation as future work.Source: §6 · p. 11

Output diversity

  • DMD produces high-quality samples but reduces output diversity, a limitation associated with reverse KL-based distribution matching.The paper suggests EM-Distillation and Score Implicit Matching as possible alternatives.Source: §6 · p. 11

Streaming speed

  • The current implementation generates around 10 FPS, while compilation, quantization, and parallelization could potentially enable real-time performance.These optimizations are presented as potential improvements rather than demonstrated results.Source: §6 · p. 11

A. VBench-Long Leaderboard Results

CausVid achieves the highest reported VBench-Long total score across the full benchmark and performs strongly across multiple evaluation dimensions. The accompanying figures and table present its comparative performance and the interface used for preference evaluation.

Leaderboard results

  • 84.27 is CausVid’s highest total score on VBench-Long, evaluated across all 946 prompts and 16 standardized metrics.The method leads several metrics, including dynamic degree, aesthetic quality, imaging quality, object class, multiple objects, and human action.Source: A. VBench-Long Leaderboard Results · p. 16
  • Figure 10 compares VBench metrics across methods and shows CausVid’s performance across different dimensions.The caption identifies CausVid in red.Source: Figure 10 · p. 17
  • Table 7 provides the full VBench-Long comparison using all 16 metrics, with the best scores shown in bold.The table is the detailed leaderboard comparison referenced by the results discussion.Source: Table 7 · p. 17

User preference study

  • Figure 11 shows a user-preference-study interface with videos from different methods randomized between left and right positions.The randomized arrangement supports comparison of displayed video outputs.Source: Figure 11 · p. 18

Related papers

Loading 2412.07772v4…