PaperlayerView exact revision on arXiv

Source-linked AI summary

Streaming Autoregressive Video Generation via Diagonal Distillation

Jinxiu Liu, Xuanming Liu, Kangfu Mei, Yandong Wen, Ming-Hsuan Yang, Weiyang Liu

arXiv:2603.09488v2cs.CV

TL;DR

Real-time streaming video generation is constrained by the temporal limitations and computational cost of existing diffusion and distillation methods. Diagonal Distillation exploits dependencies across video chunks and denoising steps, generating a 5-second video in 2.61 seconds at up to 31 FPS with a 277.3× speedup over the undistilled model.

  • Problem

    Existing video distillation methods inadequately capture temporal coherence and long-range dependencies while reducing denoising computation for real-time generation.

  • Method

    Diagonal Distillation jointly models video chunks and denoising steps through asymmetric progressive step reduction, diagonal forcing, and flow-aware temporal matching.

  • Results

    277.3× speedup over the undistilled model is achieved while generating a 5-second video in 2.61 seconds, up to 31 FPS.

  • Takeaways & Limitations

    The framework provides an efficient autoregressive video-generation strategy that balances quality and efficiency while improving dynamic consistency and reducing long-term error accumulation.

  • Takeaways & Limitations

    The technology has dual-use risks because it could be misused to create misleading content or deepfake videos.

Abstract

from arXiv · show

Large pretrained diffusion models have significantly enhanced the quality of generated videos, and yet their use in real-time streaming remains limited. Autoregressive models offer a natural framework for sequential frame synthesis but require heavy computation to achieve high fidelity. Diffusion distillation can compress these models into efficient few-step variants, but existing video distillation approaches largely adapt image-specific methods that neglect temporal dependencies. These techniques often excel in image generation but underperform in video synthesis, exhibiting reduced motion coherence, error accumulation over long sequences, and a latency-quality trade-off. We identify two factors that result in these limitations: insufficient utilization of temporal context during step reduction and implicit prediction of subsequent noise levels in next-chunk prediction (i.e., exposure bias). To address these issues, we propose Diagonal Distillation, which operates orthogonally to existing approaches and better exploits temporal information across both video chunks and denoising steps. Central to our approach is an asymmetric generation strategy: more steps early, fewer steps later. This design allows later chunks to inherit rich appearance information from thoroughly processed early chunks, while using partially denoised chunks as conditional inputs for subsequent synthesis. By aligning the implicit prediction of subsequent noise levels during chunk generation with the actual inference conditions, our approach mitigates error propagation and reduces oversaturation in long-range sequences. We further incorporate implicit optical flow modeling to preserve motion quality under strict step constraints. Our method generates a 5-second video in 2.61 seconds (up to 31 FPS), achieving a 277.3x speedup over the undistilled model.

1 INTRODUCTION

The paper introduces DiagDistill, a flow-aware diagonal distillation framework for streaming autoregressive video generation that jointly exploits temporal and denoising-step context. Its asymmetric denoising, diagonal training, and flow modeling target exposure bias, long-term error accumulation, and motion degradation while substantially reducing latency.

  • Motivation: Existing video distillation methods often underperform because they insufficiently model temporal dependencies and inter-frame consistency, leaving multi-step sampling necessary for high-quality generation.Autoregressive models suit streaming but traditional GPT-style approaches can have limited visual quality, while diffusion-transformer designs require the entire video chunk.
  • Method: DiagDistill jointly incorporates temporal and denoising-step context through diagonal attention and a diagonal denoising trajectory.Earlier chunks receive more denoising steps, while later chunks receive progressively fewer and inherit contextual structural priors.
  • Method: Diagonal Forcing simulates diagonal denoising paths during training with controlled noise injection to mitigate long-term error accumulation.The method leverages preceding chunks’ denoising trajectories as contextual priors.
  • Method: Flow Distribution Matching enhances dynamic consistency and smooth motion transitions by adding explicit temporal modeling to the distillation loss.It addresses motion degradation and amplitude attenuation in later chunks.
  • Results: 2.61 seconds generates a 5-second video, reaching up to 31 FPS and a 277.3× speedup over the undistilled model.These results are reported as state-of-the-art video-generation performance.

2 RELATED WORK

Related work spans deterministic diffusion distillation, scalable diffusion and autoregressive video generation, while Diagonal Distillation jointly models temporal and denoising-step dependencies. Its diagonal forcing, progressive step reduction, and flow distribution matching target efficient sampling with temporal and motion coherence.

  • Diffusion Distillation: Deterministic diffusion distillation regresses noise-to-sample mappings but can produce blurry few-step outputs and often requires multiple steps for acceptable quality.Examples include progressive distillation, consistency distillation, and rectified flow.
  • Autoregressive, Diffusion, and Hybrid Video Generation: Video generation primarily uses diffusion models with bidirectional attention or autoregressive models that sequentially predict spatiotemporal tokens.These represent parallel denoising and sequential next-token prediction as the dominant generation paradigms.
  • Diagonal Distillation: Figure 3 illustrates progressive step reduction from five denoising steps for the first chunk to two steps by Chunk 7, with fixed two-step denoising for chunks k ≥4.The method reuses the Key-Value (KV) cache from the preceding chunk’s final noisy frame to preserve temporal coherence and minimize latency.
  • Diagonal Distillation: Diagonal Distillation applies diagonal attention jointly across time and denoising steps to improve computational efficiency while maintaining temporal coherence.The framework addresses the limitations of treating temporal dependencies independently from denoising-step reduction.
  • Diagonal Distillation: Diagonal forcing simulates diagonal denoising paths for self-conditioned generation, while flow distribution matching enforces motion consistency with fewer denoising steps.Together, these components address temporal conditioning and motion quality under step constraints.

3 THE DIAGONAL DISTILLATION FRAMEWORK

The framework extends DMD for video by reducing denoising steps across chunks while explicitly propagating diagonal contextual priors to preserve temporal coherence and limit error accumulation. It additionally matches motion distributions using a lightweight latent-based feature module.

  • DMD’s regression loss preserves per-frame quality but does not explicitly model temporal coherence or long-range dependencies, degrading video fluidity and consistency.
  • Diagonal Denoising: Diagonal Denoising assigns more denoising steps to earlier chunks and progressively fewer to later chunks, improving the quality–efficiency trade-off through contextual priors.
  • Diagonal Denoising: For k ≥4, the method employs efficient two-step denoising using conditioning derived from previous chunks.
  • Diagonal Forcing: Diagonal Forcing injects controlled noise during training and conditions each chunk on the previous chunk’s final noised state to maintain coherence and mitigate error accumulation.
  • Flow Distribution Matching: The motion module extracts local patterns from consecutive latent-frame differences with convolution, avoiding external pretrained optical-flow estimators.The framework sets λspatial = 4 and λflow = 4 while jointly minimizing motion-distribution divergence and maintaining spatial fidelity.

4 EXPERIMENTS AND RESULTS

Experiments show that DiagDistill substantially accelerates video generation while preserving competitive visual quality, and that its asymmetric denoising, noisy-cache conditioning, and flow loss are important for quality and motion. Long-video evaluations further report stable perceptual quality and superior overall visual quality, text faithfulness, and long-term consistency.

  • Overall Comparison: 277.3× speedup over Wan2.1 is achieved while maintaining competitive visual quality at 85.26 vs. 85.3, with 1.53× lower latency than Self-Forcing.Speed tests assessed throughput and first-frame latency on a single NVIDIA H100 GPU.
  • Key Components: Diagonal Denoising assigns more steps to early chunks and fewer to later chunks, while Diagonal Forcing conditions generation on noisy rather than clean KV-cache frames.Removing either flow distribution matching loss or Diagonal Forcing reduces performance; with Diagonal Denoising removed, performance is comparable to the Self-Forcing inference cost while DiagDistill retains a 1.53× speedup.
  • Diagonal Forcing Timesteps: 100 diagonal-forcing timesteps achieve optimal temporal quality, frame quality, and text alignment, while performance degrades toward 1000 or 0 timesteps.Higher noise blurs structural priors and reduces motion, whereas lower noise can cause over-denoising and oversaturated outputs.
  • Flow Loss Weight: A flow distribution matching loss weight of 1.0 provides optimal performance by balancing motion guidance with DMD objectives.The loss primarily benefits few-step denoising by aligning it with the many-step baseline, with limited gains in many-step settings.
  • Denoising Configurations: Configuration 5333333 achieves the highest quality, whereas 4222222 offers the maximum throughput among six evaluated denoising configurations.The configurations trade video quality against computational efficiency; the selected configuration text is truncated in the supplied passage.
  • Long Video Generation: In a user study with 93 participants and 150 comparisons per model pair, DiagDistill maintained consistent perceptual quality and surpassed baselines in visual quality, text faithfulness, and long-term consistency.Baselines exhibited rapid quality decay attributed to error accumulation, while the evaluation used the first 50 MovieGenBench prompts.

5 CONCLUDING REMARKS

Diagonal Distillation enables efficient autoregressive video generation by modeling temporal dependencies across video chunks and denoising steps. Its asymmetric denoising, temporal trajectory modeling, and optical-flow distribution alignment improve motion coherence and long-range stability under reduced step budgets.

  • Core framework: Diagonal Distillation explicitly accounts for temporal structure across both video chunks and denoising steps in autoregressive video generation.The framework is presented as a method for efficient generation that exploits dependencies across these two dimensions.
  • Asymmetric denoising: Allocating more denoising steps to early chunks and progressively fewer to later ones reduces total steps without degrading motion coherence or visual fidelity.Early chunks are treated as more important for establishing global motion and appearance.
  • Temporal stability: Diagonal Forcing models the denoising trajectory along the temporal dimension, reducing error accumulation, training–inference mismatch, and drift during extended generation.The stated result is more stable long-range synthesis.
  • Motion consistency: Flow Distribution Matching aligns optical flow distributions of generated video to enforce dynamic consistency under strict step constraints.This component is introduced as an additional mechanism for preserving temporal dynamics when denoising steps are limited.

ETHICS STATEMENT · Appendix · B NOISE SCHEDULING AND MODEL PARAMETERIZATION

The study acknowledges the technology’s dual-use risks and commits to safeguards for open-sourced code and models. Its noise scheduling and parameterization follow Wan 2.1 design principles, using Flow Matching, an offset factor of 5.0, base-model preconditioning, and Diagonal Denoising settings.

  • ETHICS STATEMENT: 277.3× speedup over the baseline model motivates safeguards against misleading content and deepfake videos.The authors commit to embedding usage guidelines and restrictions when open-sourcing the code and models, while advocating traceability technologies.
  • Appendix · B NOISE SCHEDULING AND MODEL PARAMETERIZATION: Flow Matching follows the design principles of the Wan 2.1 series.The framework uses a defined time step offset.
  • Appendix · B NOISE SCHEDULING AND MODEL PARAMETERIZATION: The time step offset uses the specified t′(k, t) transformation.The appendix defines the offset expression explicitly as t′(k, t) = (k · t/1000) 1 + (k −1)(t/1000) × 1000.
  • Appendix · B NOISE SCHEDULING AND MODEL PARAMETERIZATION: The offset factor is k = 5.0.The forward process is defined after setting this factor.
  • Appendix · B NOISE SCHEDULING AND MODEL PARAMETERIZATION: The time variable satisfies t ∈[0, 1000].The data prediction model is introduced over this time range.
  • Appendix · B NOISE SCHEDULING AND MODEL PARAMETERIZATION: The data prediction model is Gθ(x, t, c) = cskip · ε −cout · vθ(cin · xt, cnoise(t′), c).This expression specifies the model’s dependence on the preconditioned noisy input, conditioning, and shifted noise time.
  • Appendix · B NOISE SCHEDULING AND MODEL PARAMETERIZATION: Preconditioning matches the base model with cskip = cin = cout = 1 and cnoise(t) = t.The coefficients are maintained identically to the base model configuration.
  • Appendix · B NOISE SCHEDULING AND MODEL PARAMETERIZATION: DMD training uses Diagonal Denoising with a time step list of [1000, 100].Time step wrapping is controlled by the warp_denoising_step parameter.

C COMPARISON OF TEMPORAL TRAINING STRATEGIES AND MODEL ARCHITECTURE DETAILS · D PSEUDO-CODE FOR DIAGONAL DENOISING WITH NOISY KV CACHE · E DETAILED ANALYSIS OF ACCELERATION AND STEP ALLOCATION

The appendices compare temporal training strategies, present Diagonal Denoising with Noisy KV Cache for efficient long-video generation, and discuss acceleration through progressive denoising-step reduction. Diagonal Forcing combines clean and generated temporal context in a diagonal conditioning pattern, while extension chunks use two-step denoising and cached outputs.

  • C COMPARISON OF TEMPORAL TRAINING STRATEGIES AND MODEL ARCHITECTURE DETAILS: Teacher Forcing uses only ground-truth previous frames, creating train-inference discrepancy and error compounding during long sequences.The comparison identifies Teacher Forcing as a clean but potentially fragile training strategy for long autoregressive generation.
  • C COMPARISON OF TEMPORAL TRAINING STRATEGIES AND MODEL ARCHITECTURE DETAILS: Diffusion Forcing exposes the model to noisy diffusion latents to enhance robustness, while Self Forcing conditions on the model’s own predictions.These strategies are included as standard baselines in the temporal-training comparison.
  • C COMPARISON OF TEMPORAL TRAINING STRATEGIES AND MODEL ARCHITECTURE DETAILS: Diagonal Forcing conditions current-frame denoising on clean ground-truth and previously generated frames arranged diagonally across time.Recent conditioning frames are model predictions from earlier autoregressive steps.
  • C COMPARISON OF TEMPORAL TRAINING STRATEGIES AND MODEL ARCHITECTURE DETAILS: Figure 10 compares Teacher Forcing, Diffusion Forcing, Self Forcing, and Diagonal Forcing for autoregressive video generation with Causal DiT.The figure depicts causal dependencies across sequential frame generation and distinguishes the strategies by their conditioning inputs.
  • D PSEUDO-CODE FOR DIAGONAL DENOISING WITH NOISY KV CACHE: Algorithm 1 processes video chunks sequentially using progressive step reduction and noisy KV-cache reuse.During the Base Phase (k ≤ 4), chunks use s[k] steps, reducing from 5 to 2, with an intermediate latent cached at the penultimate step.
  • D PSEUDO-CODE FOR DIAGONAL DENOISING WITH NOISY KV CACHE: 2 steps are used in the Extension Phase (k > 4), with each chunk conditioned on the previous chunk’s output and its first-step result cached.The stated objective is to maintain coherence while improving long-video generation efficiency.
  • E DETAILED ANALYSIS OF ACCELERATION AND STEP ALLOCATION: The acceleration analysis examines how Diagonal Forcing achieves efficiency through acceleration mechanisms and alternative denoising-step allocations.The supplied passage frames this section as a comprehensive analysis of acceleration and step-allocation strategies.

E.1 ACCELERATION ANALYSIS

Diagonal Forcing accelerates streaming generation through four synergistic optimizations: fewer denoising steps, noisy-latent KV caching, a smaller attention window, and efficient Tiny VAE tokenization. Under identical NFE budgets, it maintains or improves quality while reducing latency and increasing throughput.

  • Reduction in Denoising Steps: Fewer denoising steps reduce total Noise Function Evaluations while preserving comparable or better quality.Controlled experiments isolate this contribution from the other optimizations by matching NFE budgets.
  • Comparative efficiency: Under identical NFE budgets, Diagonal Forcing consistently outperforms Self-Forcing across quality metrics while delivering lower latency and higher throughput.Table 5 reports superior efficiency across all tested configurations.
  • Efficient KV Cache Mechanism: Noisy-latent KV caching replaces redundant clean-frame cache computations while conditioning subsequent chunks and progressing denoising.The method performs KV caching directly on the noisy latent.
  • Optimized Attention Window Size: Reducing the rolling KV cache from 6 chunks to 4 chunks lowers context overhead while maintaining seamless rolling forward.Scaling analysis indicates performance plateaus around window sizes of 12-27, with smaller windows improving latency-memory tradeoffs.
  • Efficient Tokenization Mechanism: Tiny VAE tokenization eliminates the VAE bottleneck and reduces decoding time by more than 10×.Tiny VAE encodes frames into tokens directly from latent representations and uses substantially fewer parameters than Full VAE.

E.2 STEP ALLOCATION STRATEGY ANALYSIS

The analysis compares denoising-step schedules emphasizing early frames with non-monotonic or dynamic allocations. It finds that four initial-chunk steps approach the framework’s performance upper bound, while a simple monotonically decreasing schedule remains the most effective rule.

  • Schedule Comparison: The study systematically evaluates schedules that emphasize early frames and schedules using non-monotonic or dynamic allocations.Non-monotonically decreasing schedules use average latency for reference only, marked with an asterisk.
  • Performance Upper Bound: 4 steps for the initial chunk (400000) essentially reaches the performance upper bound of the distillation framework.Using 5 or more initial steps offers diminishing returns.
  • Performance Upper Bound: Strategy 5422222 yields only marginally better metrics than 4322222 while incurring higher latency.The result supports using 4 initial steps as sufficient for the DMD framework.
  • Final Schedule Rule: Algorithmic schedule selection confirms the robustness of the manually tuned strategy and supports a monotonically decreasing allocation cyclically extended for longer videos.The schedule is identified as the most effective and principled rule.

F TECHNICAL DETAILS OF THE STREAM PROTOCOL

The streaming protocol uses 3-frame chunks with rolling KV-cache context, Tiny VAE tokenization, and pre-fill elimination for subsequent chunks. The selected 12-frame configuration balances score, latency, and memory, while Tiny VAE reduces decoding time by over 10×.

  • Chunk Size: 3-frame chunks achieve a total score of 84.48, compared with 84.29 for 1-frame chunks, while providing higher throughput.The chunk size was selected in the teacher model’s VAE latent space as the optimal trade-off.
  • Overlap, Buffering, and KV Cache Size: 12 frames balance a total score of 84.3, latency of 0.46s, and memory usage of 17.5GB.The fixed-size KV cache maintains context from the most recent 4 chunks.
  • Tokenization: Tiny VAE reduces decoding time over 10× while using significantly fewer parameters than Full VAE.Frames are encoded directly as tokens in the tiny VAE latent space.
  • Pre-fill Cost: Subsequent chunks eliminate pre-fill cost by leveraging the rolling cache, with computation focused solely on denoising new frames.The architecture has O(TL) time complexity and was validated on sequences exceeding 45 seconds without measurable performance degradation.

G DETAILED ABLATION STUDY ON MOTION FLOW FIELD REPRESENTATION … COMPARATIVE FRAMEWORK

The study evaluates a lightweight latent-space motion representation and its training requirements, then compares generated videos through a controlled, double-blind user study spanning visual quality, text faithfulness, and long-term consistency.

  • G DETAILED ABLATION STUDY ON MOTION FLOW FIELD REPRESENTATION: The method replaces external optical-flow estimators with a lightweight, learnable motion feature extractor operating directly in the diffusion model’s latent space.The module uses an EMA framework for the student and target components.
  • G DETAILED ABLATION STUDY ON MOTION FLOW FIELD REPRESENTATION: Four motion representations are examined: latent differences, latent correlations, DCT frequency components, and a learnable two-layer MLP.The representations respectively encode frame differences, pairwise correlations, low/high-frequency components, or adaptively learned features.
  • G DETAILED ABLATION STUDY ON MOTION FLOW FIELD REPRESENTATION: EMA updates the target network from the gradient-trained student, providing stable target evolution and helping prevent feature collapse.The target parameters follow θ−←µ · θ−+ (1 −µ) · θ.
  • G DETAILED ABLATION STUDY ON MOTION FLOW FIELD REPRESENTATION: The flow module adds almost negligible computation through a Conv-MLP with only two convolutional layers.This avoids the memory overhead of VAE frame decoding followed by a separate optical-flow estimator.
  • G DETAILED ABLATION STUDY ON MOTION FLOW FIELD REPRESENTATION: The regression loss is essential for stable optimization, while excessively increasing the flow-loss weight degrades performance.Without regression loss, training is highly prone to collapse; the ablation indicates that an overly strong flow constraint is detrimental.
  • H.1 EXPERIMENTAL SETUP: 93 participants evaluated long-video generation in a double-blind study using 150 comparisons per model pair across the first 50 MovieGenBench prompts.The design produced statistically significant findings and covered diverse scenarios.
  • EVALUATION CRITERIA: The evaluation assessed overall visual quality, text faithfulness, and long-term consistency across sequences of up to 45 seconds.Criteria covered visual fidelity and artifacts, semantic alignment with prompts, and quality maintenance over extended sequences.
  • COMPARATIVE FRAMEWORK: Paired comparisons tested the method against five baselines using identical prompts, controlled conditions, varied prompt complexity, and multiple scene transitions.These conditions were intended to test generalization capability and dynamic prompting performance.

H.3 USER STUDY RESULTS … H.5 DETAILED QUESTIONNAIRE EXAMPLE

The user study found statistically significant preference for the proposed method across all five baseline comparisons. Its questionnaire assessed participant demographics, video quality, comparative performance, long-term consistency, dynamic prompting, and qualitative feedback.

  • H.3 USER STUDY RESULTS: 66.1% preference for our method versus Causvid, compared with 33.9% for baseline.
  • H.3 USER STUDY RESULTS: 62.7% preference for our method versus WAN2.1, compared with 37.3% for baseline.
  • H.3 USER STUDY RESULTS: 57.9% preference for our method versus SkyReels-V2, compared with 42.1% for baseline.
  • H.3 USER STUDY RESULTS: 54.2% preference for our method versus MAGI-1 and 59.3% versus Self-Forcing, with all preference rates significant at p < 0.01.
  • PARTICIPANT DEMOGRAPHICS: 93 participants represented computer science, AI/ML, visual arts, and other backgrounds, with beginner, intermediate, and expert experience levels.Backgrounds were distributed as Computer Science (42%), AI/ML (28%), Visual Arts (18%), and Other (12%); experience levels were Beginner (15%), Intermediate (45%), and Expert (40%).
  • QUESTIONNAIRE STRUCTURE: Participants reported an average familiarity with video generation of 3.8/5.0, and the questionnaire collected professional background, expertise, and prior technology experience.
  • QUESTIONNAIRE DESIGN AND IMPLEMENTATION: Individual video quality used a 5-point Likert scale covering visual realism, color consistency, motion naturalness, artifact severity, and comparative temporal quality.Comparative tasks used side-by-side forced-choice judgments, including text alignment, temporal consistency, and quality at 15s, 30s, and 45s.

H.6 STATISTICAL ANALYSIS AND INTERPRETATION · I SOCIETAL IMPACT CONSIDERATIONS · J PROMPTS

The paper reports statistical analyses supporting superior visual quality and temporal consistency, especially in long videos, while acknowledging both misuse risks and broad prompt scenarios. Its prompts span cinematic scenes, dynamic events, sequential narratives, and varied environments.

  • H.6 STATISTICAL ANALYSIS AND INTERPRETATION: The user study applied preference-rate confidence intervals, paired t-tests, ANOVA, and thematic coding for statistical and qualitative analysis.These methods were enabled by the comprehensive user-study design.
  • H.6 STATISTICAL ANALYSIS AND INTERPRETATION: 66.1% was the highest preference rate against Causvid, highlighting stronger visual quality and temporal consistency during long video generation.The result is attributed to addressing error accumulation affecting baseline methods.
  • I SOCIETAL IMPACT CONSIDERATIONS: Lowering computational costs for real-time video generation may increase misuse risks, including deepfake-enabled misinformation, despite potential positive creative applications.The societal-impact discussion presents both risks and beneficial uses of generative video technology.
  • J PROMPTS: Figure 4 prompts cover stylized reading, beach, flood, nighttime traffic, cycling, and robotic glacier scenes with explicit camera motion and atmosphere.The scenarios combine diverse subjects, environments, motion patterns, and cinematic descriptions.
  • J PROMPTS: Figure 7 prompts define ten-shot narratives involving a lighthouse storm, racing-car recovery, market sand spill, mountain hiking, cycling, and coral-reef life.These sequences specify shot progression, evolving events, camera viewpoints, and environmental changes.
  • J PROMPTS: Figure 9 prompts present three sequential videos: purposeful walking across locations, orderly room sweeping, and a foggy coastal telephone booth.The prompts emphasize continuity through repeated subjects, staged actions, and gradual environmental or narrative transitions.
Loading 2603.09488v2…