Source-linked AI summary
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion
Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, Eli Shechtman
TL;DR
Self Forcing addresses exposure bias in autoregressive video diffusion, where training uses ground-truth context but inference conditions on self-generated outputs. It trains through autoregressive self-rollouts with holistic video-level distribution matching, achieving real-time generation with competitive or superior quality against slower alternatives.
Problem
Exposure bias creates a train-test distribution gap because models trained on ground-truth context must generate using their own imperfect outputs at inference.
Method
Self Forcing unrolls autoregressive generation during training with KV caching, matches complete video-sequence distributions, and uses few-step diffusion with gradient truncation for efficiency.
Results
17 FPS with sub-second latency on a single H100 GPU, while achieving competitive or superior quality compared with recent slow bidirectional and autoregressive video diffusion models.
Takeaways & Limitations
Self Forcing supports interactive video generation and maintains consistent quality across chunk-wise and frame-wise autoregressive setups while outperforming baselines.
Takeaways & Limitations
Quality degradation remains when generating videos substantially longer than the training context, and gradient truncation may limit learning long-range dependencies.
Abstract
from arXiv · showhide
We introduce Self Forcing, a novel training paradigm for autoregressive video diffusion models. It addresses the longstanding issue of exposure bias, where models trained on ground-truth context must generate sequences conditioned on their own imperfect outputs during inference. Unlike prior methods that denoise future frames based on ground-truth context frames, Self Forcing conditions each frame's generation on previously self-generated outputs by performing autoregressive rollout with key-value (KV) caching during training. This strategy enables supervision through a holistic loss at the video level that directly evaluates the quality of the entire generated sequence, rather than relying solely on traditional frame-wise objectives. To ensure training efficiency, we employ a few-step diffusion model along with a stochastic gradient truncation strategy, effectively balancing computational cost and performance. We further introduce a rolling KV cache mechanism that enables efficient autoregressive video extrapolation. Extensive experiments demonstrate that our approach achieves real-time streaming video generation with sub-second latency on a single GPU, while matching or even surpassing the generation quality of significantly slower and non-causal diffusion models. Project website: http://self-forcing.github.io/
1 Introduction
Autoregressive video generation supports low-latency streaming but suffers from exposure bias and accumulating errors. Self Forcing trains on self-generated context with video-level distribution matching, achieving efficient real-time generation while retaining strong quality.
- Bidirectional video diffusion generates all frames simultaneously, limiting real-time streaming because future information is unavailable when producing the current frame.
- Autoregressive models generate frames sequentially and reduce viewing latency, but often sacrifice visual fidelity because of lossy vector quantization.
- Teacher Forcing and Diffusion Forcing condition denoising on ground-truth or noisy context, leaving training outputs mismatched with inference-time generations.
- Self Forcing autoregressively rolls out self-generated frames during training and applies holistic distribution matching to complete output videos.
- A few-step diffusion backbone, gradient truncation, and rolling KV caching make sequential post-training efficient and improve video extrapolation.
- 17 FPS with sub-second latency on one H100 GPU is achieved alongside competitive or superior quality versus slower bidirectional and autoregressive diffusion models.
2 Related Work
Related video-generation work includes GANs, diffusion, autoregressive, hybrid, and rolling-diffusion approaches. Self Forcing is positioned most closely against CausVid by targeting distribution mismatch in few-step autoregressive diffusion training.
- GANs for Video Generation: GAN video generators inherently avoid exposure bias because their training and inference generation processes are the same.
- Autoregressive/Diffusion Models for Video Generation: Video diffusion models typically denoise all frames simultaneously with bidirectional attention, whereas autoregressive models generate spatiotemporal tokens sequentially.
- Autoregressive-Diffusion Hybrid Models: Hybrid autoregressive-diffusion models use long temporal and spatial prediction chains that can accumulate substantial errors.
- Rolling Diffusion and Variants: Rolling-diffusion methods support sequential long-video generation but do not strictly follow autoregressive chain-rule decomposition and can incur significant latency.
- CausVid: CausVid trains few-step autoregressive diffusion with Diffusion Forcing and DMD, but its training outputs differ from the inference distribution targeted by DMD.
3 Self Forcing: Briding Train-Test Gap via Holistic Post-Training
Self Forcing trains autoregressive video diffusion through self-generated rollouts and holistic video-level distribution matching, aligning training with inference. A few-step backbone, stochastic gradient truncation, and rolling KV caching make this approach efficient for training and long-video extrapolation.
- 3.1 Preliminaries: Autoregressive Video Diffusion Models: Autoregressive video diffusion factorizes the video distribution into frame-wise conditionals, with each frame generated by iterative diffusion denoising conditioned on previous frames.Frames may also be generated in chunks, which are denoted as frames for simplicity.
- 3.1 Preliminaries: Autoregressive Video Diffusion Models: Teacher Forcing and Diffusion Forcing train frame-wise denoising objectives using clean or noisy context frames, respectively, with causal attention implemented through specialized masks.Teacher Forcing typically shares timesteps across frames, whereas Diffusion Forcing samples them independently.
- 3.2 Autoregressive Diffusion Post-Training via Self-Rollout: Self Forcing performs autoregressive self-rollout during training, generating each frame from previously self-generated outputs while using KV caching as in inference.The approach applies iterative denoising to each conditional distribution and caches information from prior generated frames.
- 3.2 Autoregressive Diffusion Post-Training via Self-Rollout: A few-step diffusion model and stochastic gradient truncation reduce training cost by backpropagating only through each frame’s final denoising step and randomly sampling the final step.The method uses a sampled denoising step for each sequence and disables gradients through earlier denoising steps.
- 3.3 Holistic Distribution Matching Loss: Self Forcing matches the holistic distribution of complete generated videos to the data distribution instead of matching frame-wise conditionals from data-sampled contexts.This changes training contexts from clean or noisy data-distribution samples to samples from the model distribution.
- 3.4 Long Video Generation with Rolling KV Cache: The rolling KV cache stores tokens from the latest L frames, evicts the oldest entries when full, and enables endless extrapolation without KV recomputation at O(TL) time complexity.This avoids the O(TL2) cost of bidirectional sliding-window inference and the O(L2 + TL) cost of prior causal implementations.
4 Experiments
Experiments show that Self Forcing combines strong video quality with real-time autoregressive generation and efficient training. It also remains robust across distribution-matching objectives and supports high-throughput video extrapolation.
- Main results: 17.0 FPS with sub-second latency accompanies the highest VBench scores among compared models.The chunk-wise model is also consistently preferred over all evaluated alternatives in the user study.
- Main results: Self Forcing is consistently preferred over all baselines in the human preference study.
- Ablations: Self Forcing consistently outperforms alternative training approaches across DMD, SiD, and GAN distribution-matching objectives.
- Ablations: Self Forcing maintains consistent quality from chunk-wise to frame-wise autoregressive generation, unlike baselines that degrade with longer autoregressive unrolling.Baseline degradation commonly appears as progressive over-saturation or over-sharpening.
- Video extrapolation: Rolling KV cache reaches 16.1 FPS while mitigating artifacts, compared with 4.6 FPS when recomputing the cache for 10-second videos.Training without access to the initial image latent is used to reduce rolling-cache artifacts.
- Training efficiency: Self Forcing achieves comparable per-iteration time to Teacher Forcing and Diffusion Forcing while attaining superior quality under equal wall-clock budgets.Each DMD experiment converges in approximately 1.5 hours on 64 H100 GPUs.
5 Discussion
The discussion frames Self Forcing as a response to the mismatch introduced by parallelizable training and as an integration of autoregressive, diffusion, and GAN ideas. It also identifies limits in extrapolating beyond the training context and learning long-range dependencies.
- Fundamental limitation of the parallelizable training paradigm: Parallelizable training can misalign the training and inference distributions in sequential state-tracking problems.
- Interplay between AR, Diffusion, and GANs: Autoregressive and diffusion models offer complementary chain-rule and latent-variable factorizations that can be composed in a nested manner.The discussion connects this composition with GAN-style distribution matching.
- Limitation and future directions: Quality degradation remains observable when generating videos substantially longer than those used during training.
- Limitation and future directions: Gradient truncation supports memory efficiency but may limit learning of long-range dependencies.Future directions include improved extrapolation and recurrent architectures such as state-space models.
A Implementation Details
The implementation combines flow matching, few-step diffusion, distribution-matching objectives, and GPU-efficient training configurations. It also specifies the model parameterization, data preprocessing, and stability choices used in experiments.
- Implementation: The implementation uses Wan2.1 and CausVid code, FlexAttention for TF and DF baselines, and FlashAttention-3 for Self Forcing.These attention implementations distinguish the baseline and Self Forcing systems.
- Model and noise schedule: The model uses flow matching with a shift factor of 5 and a four-step diffusion schedule [1000, 750, 500, 250].The data-prediction parameterization keeps cskip = cin = cout = 1 and cnoise(t) = t.
- Data preprocessing: Prompt preprocessing filters short, command-containing, and unsafe prompts, yielding approximately 250k prompts from the VidProS subset.The source subset contains around 1M semantically unique user-written text-to-video prompts.
- Training configuration: DMD converges in approximately 1.5 hours, while SiD and GAN training take 2–3 hours on 64 H100 GPUs.Most runs use 64 80GB NVIDIA GPUs with per-GPU batch size 1, plus gradient accumulation when needed.
- Distribution matching: Training uses DMD, SiD, or GAN distribution-matching objectives with pretrained real-score and critic networks.The GAN setup adds cross-attention layers and classification heads, while SiD commonly adopts α = 1 for stability.
B Importance of local attention training in rolling KV cache
Local attention window training improves rolling KV cache-based video extrapolation beyond the training context. The naive baseline instead produces severe visual artifacts.
- Rolling KV cache extrapolation: The naive baseline exhibits severe visual artifacts when extrapolating video beyond the training context length.Its chunks always attend to the first chunk during denoising.
- Rolling KV cache extrapolation: Restricting the attention window prevents the last chunk from attending to the first chunk during denoising.This is the proposed local attention window training setting used with rolling KV cache extrapolation.
C VBench Scores Across All Dimensions
Across all 16 VBench metrics, Self Forcing shows strong semantic alignment and frame-wise quality, with a motion–temporal-consistency trade-off between its AR variants.
- Semantic alignment: Self Forcing generally outperforms representative models on semantic alignment, especially scene, object class, multiple objects, and human action metrics.The comparison evaluates both chunk-wise and frame-wise AR versions against representative models.
- Frame-wise quality: Self Forcing achieves good frame-wise quality on aesthetic quality and imaging quality metrics.These results are reported in the all-16-metric VBench evaluation.
- AR variant comparison: The frame-wise AR variant has more dynamic motion but worse background consistency, motion smoothness, and temporal flickering than the chunk-wise variant.The reported comparison indicates a trade-off between dynamic degree and temporal consistency.
D Broader Societal Impact
The paper evaluates Self Forcing against representative video-generation systems and discusses both beneficial applications and societal risks. Real-time generation reduces computational barriers that may otherwise limit misuse.
- Evaluation context: Figure 8 compares Self Forcing with SkyReels-V2, Wan2.1-1.3B, MAGI-1, and CausVid using all 16 VBench metrics.The visualization is intended to compare performance across the full VBench metric set.
- Risks: Video generative models can enable disinformation through deepfakes and amplify harmful stereotypes or societal biases without responsible governance.These risks are identified as significant potential societal consequences of generative video modeling.
- Dual-use implications: Real-time video generation removes computational cost as a practical barrier to misuse, despite potential positive applications such as creative production and accessibility tools.The paper encourages detection, watermarking, and policy research to mitigate these dual-use risks.
E User Study Details
The user preference study compares paired videos generated from the same text prompt, asking users to judge overall quality and prompt alignment.
- Users compare two videos side by side for the same text prompt and select the overall better video.The judgment considers both video quality and alignment with the prompt.
- The study evaluates all 1003 prompts from MovieGenBench, with each prompt assessed by one user.
- Figure 9 provides screenshots of the instructions shown to study participants.