Source-linked AI summary
AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion
Mingzhen Sun, Weining Wang, Gen Li, Jiawei Liu, Jiahui Sun, Wanquan Feng, Shanshan Lao, SiYu Zhou, Qian He, Jing Liu
TL;DR
Video generation needs realistic, temporally coherent sequences, but existing autoregressive and synchronous diffusion approaches have limitations in training–inference consistency, error accumulation, or timestep flexibility. AR-Diffusion combines diffusion with causal attention, non-decreasing framewise timesteps, and FoPP and AD schedulers. It achieves competitive or state-of-the-art results across four benchmarks, including a reported 60.1% FVD improvement on UCF-101 over a prior asynchronous diffusion model.
Problem
Video generation must produce visually realistic and temporally coherent sequences, while existing approaches face training–inference inconsistencies, error accumulation, or rigid timestep constraints.
Method
AR-Diffusion combines diffusion during training and inference with temporal causal attention, non-decreasing framewise timesteps, and FoPP and AD timestep schedulers.
Results
60.1% FVD improvement over the previous SOTA asynchronous video diffusion model is reported on UCF-101, alongside competitive or state-of-the-art results across four benchmarks.
Takeaways & Limitations
The method supports both asynchronous and synchronous generation while allowing flexible video lengths and preserving temporal coherence.
Takeaways & Limitations
Chunked autoregressive generation can suffer temporal inconsistencies and motion discontinuities because it conditions on limited preceding context.
Abstract
from arXiv · showhide
The task of video generation requires synthesizing visually realistic and temporally coherent video frames. Existing methods primarily use asynchronous auto-regressive models or synchronous diffusion models to address this challenge. However, asynchronous auto-regressive models often suffer from inconsistencies between training and inference, leading to issues such as error accumulation, while synchronous diffusion models are limited by their reliance on rigid sequence length. To address these issues, we introduce Auto-Regressive Diffusion (AR-Diffusion), a novel model that combines the strengths of auto-regressive and diffusion models for flexible, asynchronous video generation. Specifically, our approach leverages diffusion to gradually corrupt video frames in both training and inference, reducing the discrepancy between these phases. Inspired by auto-regressive generation, we incorporate a non-decreasing constraint on the corruption timesteps of individual frames, ensuring that earlier frames remain clearer than subsequent ones. This setup, together with temporal causal attention, enables flexible generation of videos with varying lengths while preserving temporal coherence. In addition, we design two specialized timestep schedulers: the FoPP scheduler for balanced timestep sampling during training, and the AD scheduler for flexible timestep differences during inference, supporting both synchronous and asynchronous generation. Extensive experiments demonstrate the superiority of our proposed method, which achieves competitive and state-of-the-art results across four challenging benchmarks.
IA, CAS
Jiawei Liu is affiliated with Bytedance Inc.
- Jiawei Liu is affiliated with Bytedance Inc.
- The passage identifies Jiawei Liu as an author.
- The listed affiliation is Bytedance Inc.
IA, CAS UCAS
The listed authors include Siyu Zhou, Wanquan Feng, Shanshan Lao, and Qian He, with an internship footnote associated with the research.
- Siyu Zhou is affiliated with Bytedance Inc.
- Wanquan Feng is affiliated with Bytedance Inc.
- Shanshan Lao is affiliated with Bytedance Inc.
- The research was completed during an internship at Bytedance, and the paper provides a project repository and playable video samples.
- Qian He is affiliated with Bytedance Inc.
1. Introduction
The paper targets video generation that must preserve visual realism and temporal consistency while addressing limitations in asynchronous autoregressive and synchronous diffusion approaches. AR-Diffusion combines diffusion, causal attention, constrained timesteps, and specialized schedulers, achieving competitive or state-of-the-art results across four benchmarks.
- Video generation seeks visually realistic frames with temporally consistent motion.
- Asynchronous autoregressive models support variable-length generation but can accumulate errors from training–inference inconsistencies.
- Synchronous diffusion applies one timestep across frames, limiting timestep-composition flexibility.
- AR-Diffusion uses diffusion during training and inference, temporal causal attention, and non-decreasing framewise timesteps.
- FoPP balances training timestep compositions, while AD adaptively varies neighboring-frame timestep differences during inference.
- 60.1% FVD improvement over the previous SOTA asynchronous video diffusion model is reported on UCF-101.
2. Related Works
Related work spans synchronous generation with shared noise, autoregressive or chunked asynchronous generation, and asynchronous diffusion with frame-specific noise. These approaches trade temporal flexibility and computational efficiency against temporal consistency, training stability, or search-space complexity.
- Synchronous diffusion models apply consistent noise across frames to preserve uniform information entropy and temporal coherence.
- GAN-based synchronous approaches capture temporal dynamics but face mode collapse and training instability challenges.
- Chunked autoregressive diffusion reduces computation by predicting multiple frames in parallel, but limited temporal context can produce inconsistent dynamics.
- Autoregressive methods generate frames sequentially, enabling variable-length videos but risking training–inference inconsistency and error accumulation.
- Asynchronous diffusion assigns distinct noise levels to frames, improving sampling flexibility and handling temporal variability.
- Independent framewise timesteps expand the search space and can make asynchronous diffusion less stable or less competitive than synchronous methods.
3. Methods
AR-Diffusion combines latent video encoding, temporally causal decoding, and diffusion-based generation with constrained timesteps. Its FoPP training scheduler and AD inference scheduler support diverse, flexible video-generation settings.
- Framework: AR-VAE encodes videos into compact latent features, while AR-Diffusion generates videos in the latent space.The framework represents each frame with latent tokens before diffusion-based generation.
- AR-VAE: The AR-VAE decoder uses temporal causal attention so later frames can reference earlier latent tokens without reverse access.This unidirectional design strengthens temporal correlations and improves temporal consistency during reconstruction.
- AR-Diffusion: AR-Diffusion corrupts clean video tokens at sampled timesteps and trains a Transformer with temporal causal attention to predict clean tokens.The same diffusion formulation is used during training and inference, where sampling iteratively transforms noise into video.
- Timestep Constraint: The non-decreasing constraint t1 ≤ t2 ≤ ... ≤ tF keeps earlier frames at least as clear as later frames while providing more valid compositions than equal timesteps.For F = 16 and T = 1000, equal constraints permit O(1e3) compositions, whereas the non-decreasing constraint provides O(1e32).
- Schedulers: FoPP balances timestep-composition and frame-specific sampling during training, while AD adaptively controls neighboring-frame timestep differences during inference.AD regulates compositions from all frames at timestep T to all frames at timestep 0 and supports asynchronous and synchronous generation.
4. Experiments
Experiments evaluate AR-Diffusion across four datasets using quantitative, timestep, qualitative, and ablation analyses. The results show strong video quality and temporal consistency, with timestep choices affecting quality–efficiency trade-offs and key components materially affecting performance.
- Experimental Setup: Experiments use FaceForensics, Sky-Timelapse, Taichi-HD, and UCF-101, evaluating realism and temporal consistency with FID-img, FID-vid, and FVD.FVD is measured on 16- and 128-frame clips, and lower scores indicate better results.
- Quantitative Comparison: AR-Diffusion achieves lower FVD16 and FVD128 scores than other methods across the four datasets, indicating stronger video quality and temporal consistency.Reported examples include 71.9 and 265.7 on FaceForensics, 40.8 FVD16 on Sky-Timelapse, 66.3 and 376.3 on Taichi-HD, and 186.6 and 572.3 on UCF-101.
- Timestep Analysis: For 16-frame generation, moderate timestep differences often balance quality and efficiency, while larger s values tend to worsen FVD and increase inference time.The best reported settings include s = 0 for FaceForensics and Sky-Timelapse, s = 10 for Taichi-HD, and s = 15 for UCF-101.
- Timestep Analysis: For 128-frame generation, s = 5 achieves relatively lower FVD scores, supporting smaller timestep differences for longer sequences.The reported overall balance is s = 10 for 16-frame generation and s = 5 for 128-frame generation.
- Qualitative Comparison: Qualitative comparisons show AR-Diffusion producing clearer appearances, more recognizable details, and stronger motion than several prior methods.The comparisons cover UCF-101, Sky Time-lapse, and Taichi-HD examples.
- Ablation Study: Ablations show that removing FoPP raises FVD from 62.8 to 101.0, while removing temporal causal attention raises FVD to 209.8.Replacing x0 prediction with ϵ prediction raises FVD to 257.6, and removing AR-VAE improvements raises it to 148.3.
5. Conclusion
AR-Diffusion combines auto-regressive and synchronous diffusion ideas for asynchronous video generation through timestep constraints and specialized schedulers. The paper reports competitive or state-of-the-art results on four challenging benchmarks.
- Conclusion: AR-Diffusion combines auto-regressive and synchronous diffusion models through a non-decreasing timestep constraint and specialized training and inference schedulers.FoPP balances timestep-composition sampling during training, while AD regulates timestep compositions during inference.
- Conclusion: AR-Diffusion obtains competitive or state-of-the-art results on four challenging benchmarks.The conclusion presents this as the paper’s overall experimental outcome.
- Conclusion: Temporal causal attention is used because later frames contain more noise and less information, while causal processing preserves auto-regressive flexibility.The discussion also identifies reduced noise interference as a motivation.
B. Limitation
The paper identifies limited use of image data as a primary limitation and future research direction.
- Limitation: AR-Diffusion is trained with video data, leaving potential performance gains from incorporating more readily available and diverse image data.The paper suggests combining image and video data to improve visual quality and frame diversity, especially when video data is scarce.
C. Training Stability
The section compares training stability and reconstruction quality across the proposed components and datasets. AR-Diffusion’s loss behavior and AR-VAE reconstructions are evaluated qualitatively and through reported experimental settings.
- Training stability: The training-loss curves compare Diffusion Forcing on UCF-101 with AR-Diffusion on the same dataset.Figures 5 and 6 provide the corresponding loss-curve visualizations.
- Training stability: Diffusion Forcing’s training loss decreases overall but shows noticeable fluctuations and spikes around 40k and 80k steps.These spikes indicate moments of instability despite the continued downward trend.
- AR-VAE reconstruction: AR-VAE reconstructions preserve spatial details and temporal progression across FaceForensics, UCF-101, TaiChi-HD, and Sky-Timelapse.The reported reconstructions maintain visual fidelity and temporal consistency across diverse video content.
E. Ablation study on AR-VAE
The AR-VAE ablation compares VAE choices and efficiency under a fixed training setup. The proposed AR-VAE uses fewer tokens and supports larger batches, faster training and inference, and better generation performance than Open-Sora-VAE.
- VAE comparison: Open-Sora-VAE uses 8 times more tokens than AR-VAE to represent a video.The comparison uses 4 A800 GPUs for 100 hours without the fine-tune stage.
- Efficiency: AR-VAE enables a batch size of 16 versus 2 for Open-Sora-VAE, with faster training and inference.The token-count difference is reported as the reason for the smaller Open-Sora-VAE batch size and slower processing.
- Efficiency: The method shows superior efficiency when generation and decoding time are both included.The efficiency comparison uses official codes and checkpoints.
H. Samples on Long Video Generation
The section presents long-video samples and reconstructions across four datasets. AR-Diffusion generates 128-frame videos with visually realistic content and temporally coherent motion across varied scenes and actions.
- Long-video generation: AR-Diffusion generates 128-frame videos on TaiChi-HD, Sky-Timelapse, UCF-101, and FaceForensics.Each displayed video uses four skipped frames, and additional samples are referenced by the paper.
- Frame reconstruction: AR-VAE reconstructions preserve real-frame structure and details across FaceForensics, UCF-101, TaiChi-HD, and Sky-Timelapse.Figure 7 places real frames in the first row and reconstructed frames in the second row.
- Qualitative results: Generated videos maintain smooth motion and consistent details for Tai Chi, sky changes, human actions, and facial expressions.The reported samples show seamless transitions and temporal coherence across all four datasets.