PaperlayerView exact revision on arXiv

Source-linked AI summary

Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention

Chengtao Lv, Yumeng Shi, Yushi Huang, Ruihao Gong, Shen Ren, Wenya Wang

arXiv:2602.04789v4cs.CV

TL;DR

Autoregressive video diffusion remains bottlenecked by quadratic attention, while bidirectional sparse-attention policies can degrade AR generation quality. Light Forcing addresses this with chunk-aware sparsity allocation and hierarchical frame/block mask selection, consistently improving quality and latency, including an 84.5 VBench score and 1.3× end-to-end speedup on Self Forcing.

  • Problem

    Quadratic spatiotemporal attention limits efficient AR video deployment, while directly transferring bidirectional sparse-attention policies can substantially degrade quality by mishandling chunk contributions and past context.

  • Method

    Light Forcing combines Chunk-Aware Growth for progressive chunk-wise sparsity allocation with Hierarchical Sparse Attention for coarse-to-fine frame- and block-level context selection.

  • Results

    84.5 VBench score and 1.3× end-to-end speedup are achieved on Self Forcing, while experiments across AR models show consistent quality and latency gains over existing sparse-attention methods.

  • Takeaways & Limitations

    Light Forcing supports real-time AR video synthesis on consumer GPUs and provides a framework for more scalable autoregressive video generation.

  • Takeaways & Limitations

    The analysis assumes that q and k/v have the same shape.

Abstract

from arXiv · show

Advanced autoregressive (AR) video generation models have improved visual fidelity and interactivity, but the quadratic complexity of attention remains a primary bottleneck for efficient deployment. While existing sparse attention solutions have shown promise on bidirectional models, we identify that applying these solutions to AR models leads to considerable performance degradation for two reasons: isolated consideration of chunk generation and insufficient utilization of past informative context. Motivated by these observations, we propose \textsc{Light Forcing}, the \textit{first} sparse attention solution tailored for AR video generation models. It incorporates a \textit{Chunk-Aware Growth} mechanism to quantitatively estimate the contribution of each chunk, which determines their sparsity allocation. This progressive sparsity increase strategy enables the current chunk to inherit prior knowledge in earlier chunks during generation. Additionally, we introduce a \textit{Hierarchical Sparse Attention} to capture informative historical and local context in a coarse-to-fine manner. Such two-level mask selection strategy (i.e., frame and block level) can adaptively handle diverse attention patterns. Extensive experiments demonstrate that our method outperforms existing sparse attention in quality (e.g., 84.5 on VBench) and efficiency (e.g., $1.2{\sim}1.3\times$ end-to-end speedup). Combined with other efficient solutions, \textsc{Light Forcing} further achieves a $2.0{\sim}3.0\times$ end-to-end speedup across diverse GPUs (e.g., 27.4\,FPS on RTX 5090 and 33.9\,FPS on H100). Code is released via this \href{https://github.com/chengtao-lv/LightForcing}{link}.

1. Introduction

Light Forcing addresses the quadratic attention bottleneck in autoregressive video generation, where directly applying bidirectional sparse-attention methods substantially degrades quality. It combines chunk-aware sparsity allocation with hierarchical frame- and block-level selection to preserve informative context while improving efficiency.

  • Motivation: Quadratic spatiotemporal 3D full attention remains a major deployment bottleneck for autoregressive video models.For 480p generation with Self-Forcing 1.3B, attention consumes nearly three times the runtime of all other components combined at the last chunk.
  • Motivation: Directly transferring sparse-attention methods from bidirectional models substantially degrades autoregressive generation quality.The degradation is linked to accumulated autoregressive errors and insufficient accounting for heterogeneous chunk contributions.
  • Method: Chunk-Aware Growth quantitatively estimates each chunk’s contribution and allocates higher attention budgets to earlier chunks, progressively decaying them later.This chunk-aware policy reduces error propagation while preserving efficiency.
  • Method: Hierarchical Sparse Attention selects sparse masks at frame and block levels in a coarse-to-fine pipeline.The two-level strategy captures informative historical context and local dependencies while maintaining fast execution.
  • Evaluation: Across Self Forcing, LongLive, and Infinite-Forcing on VBench and VBench-Long, Light Forcing consistently outperforms existing sparse-attention approaches.The evaluation covers three autoregressive video generation models and two benchmarks.

2. Related Work

Related work distinguishes bidirectional and autoregressive video diffusion, reviews conditioning strategies for autoregressive generation, and notes that sparse-attention research has focused mainly on bidirectional models. Sparse attention for autoregressive video generation remains largely unexplored.

  • Autoregressive Video Generation: Autoregressive video models generate tokens or frames sequentially, making them inherently more suitable for real-time streaming than bidirectional models that denoise all frames jointly.Bidirectional models process all frames together, whereas autoregressive models generate the next token or frame sequentially.
  • Autoregressive Video Generation: Teacher Forcing conditions training on ground-truth tokens but suffers from reduced visual fidelity when generating long videos.Diffusion Forcing instead trains with conditioning at arbitrary noise levels and has been adopted by SkyReels-V2 and Magi-1.
  • Sparse Attention: Many methods alleviate the runtime bottleneck from quadratic-complexity attention in bidirectional video diffusion across attention types and sparsity levels.Examples include combining linear and softmax attention and using twin-level or pyramid-level sparse representations.
  • Sparse Attention: Sparse-attention exploration for autoregressive video generation remains largely uncharted.Existing sparse-attention work has primarily addressed bidirectional video diffusion models.

3. Preliminaries

This section introduces autoregressive video diffusion, which combines inter-chunk autoregression with intra-chunk diffusion, and blockwise sparse attention, which improves accelerator efficiency by skipping inactive tiles. The resulting computation scales with active blocks rather than the full quadratic sequence size.

  • Autoregressive video diffusion: Autoregressive video diffusion decomposes synthesis into inter-chunk autoregression and intra-chunk diffusion for temporal dependency modeling and high-fidelity frame generation.The joint distribution factorizes over sequential video chunks conditioned on prior clean frames and condition c.
  • Autoregressive video diffusion: Each conditional term is approximated by a few-step diffusion generator conditioned on a KV cache containing previous clean frames.The denoising process uses successive operators, with forward corruption injecting Gaussian noise at a lower noise level for the next state.
  • Blockwise sparse attention: Blockwise sparse attention partitions sequence dimensions into tiles and represents active computation with a binary block mask.For simplicity, the formulation assumes q and k/v have the same shape.
  • Blockwise sparse attention: Active tiles are computed directly without materializing the element-wise mask, while inactive tiles are skipped entirely.Dense computation is retained within each active tile, matching GPU-friendly kernels such as FlashAttention.
  • Blockwise sparse attention: Computational and memory costs scale with the number of active blocks rather than n^2.This block-granular execution improves utilization and memory access patterns on modern accelerator hardware.

4. LIGHT FORCING

Light Forcing addresses autoregressive video diffusion bottlenecks with Chunk-Aware Growth, which allocates sparsity across chunks, and Hierarchical Sparse Attention, which retrieves informative historical and local context efficiently. Together, these mechanisms preserve early visual priors, bound attention complexity, and mitigate long-range consistency degradation and history forgetting.

  • Motivation: Chunk-agnostic sparse-attention policies can fail in autoregressive video diffusion because they ignore heterogeneous contributions across generated chunks.Existing bidirectional-model acceleration methods exhibit timestep- and layer-dependent sensitivity, making direct transfer problematic.
  • Chunk-Aware Growth: 80% sparsity in the first chunk causes irreversible visual-quality loss, whereas applying it to later chunks remains nearly lossless.First-chunk sparsity produces over-saturation and exposure-bias artifacts; the first chunk therefore serves as a visual anchor whose priors later chunks can inherit.
  • Chunk-Aware Growth: Chunk-Aware Growth assigns lower sparsity to error-prone chunks by accounting for finite-step sampling and score-estimation errors.The first chunk uses dense attention, while subsequent chunk ratios depend on noise levels and a modulated factor calibrated to the target FLOPs.
  • Hierarchical Sparse Attention: Hierarchical Sparse Attention replaces growing full-history attention with coarse-to-fine retrieval of keyframes followed by dynamic sparse attention over selected frames.Its components include Token Compression, Mask Selection, and Blockwise Sparse Attention, while the two-stage design addresses history forgetting and long-video consistency.
  • Hierarchical Sparse Attention: 2% end-to-end runtime increase is the approximate overhead of HSA’s frame-retrieval step, while its attention complexity remains fixed regardless of historical-frame count.CAG operates at the macro level by allocating chunk sparsity, whereas HSA determines the attention pattern within chunks.

5. Experiments

Experiments show that LIGHT FORCING improves video-generation quality and speed over sparse-attention baselines while preserving performance against dense attention and during longer autoregressive rollouts. Ablations and deployment profiling further validate its components, robustness, and practical acceleration benefits.

  • Main Results: LIGHT FORCING outperforms sparse-attention baselines on most metrics and achieves the highest speedups: 1.3× on Self Forcing and 1.19× on LongLive.The gains include Imaging Quality and Subject Consistency.
  • Main Results: 84.5 vs. 84.1 and 83.9 vs. 83.2 in Total Score, LIGHT FORCING exceeds dense FlashAttention on Self Forcing and LongLive, respectively.This indicates sparse attention can achieve lossless performance relative to dense attention.
  • Long-Video Results: 84.1 vs. 83.6 in Total Score, LIGHT FORCING slightly exceeds Infinite-Forcing on 15-second VBench-Long generation.It improves Quality Score from 84.6 to 85.4, Imaging Quality from 68.7 to 69.5, Motion Smoothness from 98.5 to 98.6, and Dynamic Degree from 54.7 to 64.7.
  • Ablation Studies: Directly applying 1D sparse attention at 90% sparsity causes severe quality collapse, while fine-tuning recovers performance only to 82.8 versus dense attention’s 84.1 Total Score.Adding CAG produces notable gains in Aesthetic Quality and Imaging Quality.
  • Efficient Deployment: 3.79× speedup in attention time and 3.08× end-to-end speedup, LIGHT FORCING accelerates deployment using LightVAE, FP8 layers, and optimized Triton kernels.The deployment profile uses per-channel weight quantization and per-token activation quantization.

6. Conclusion

LIGHT FORCING is a sparse attention framework tailored for autoregressive video diffusion, combining chunk-aware growth with hierarchical sparse attention. Experiments show improved efficiency and generation quality, enabling real-time synthesis on consumer GPUs and supporting scalable AR video generation.

  • Framework: LIGHT FORCING is a sparse attention framework tailored for autoregressive video diffusion.It is designed specifically for AR video generation.
  • Mechanisms: Chunk-aware growth and hierarchical sparse attention mitigate error accumulation while preserving long-range context.These are the framework’s two introduced mechanisms.
  • Results: Experiments demonstrate consistent improvements in efficiency and generation quality, enabling real-time video synthesis on consumer GPUs.The conclusion presents these outcomes as a foundation for scalable AR video generation.

A. Implementation details of Baselines · B. Prompts for Long Video Generation

The paper specifies how sparse-attention baselines are adapted for autoregressive video generation, including model-specific windows, sparsity ratios, clustering settings, and Light Forcing configurations. It also lists qualitative-generation prompts for 15-second videos.

  • A. Implementation details of Baselines: STA uses 3D tiles with a (3, 3, 3) attention window for all experiments.Unlike the original setting, sparse attention is applied at every step because autoregressive models are typically few-step generators, such as 4 steps.
  • A. Implementation details of Baselines: Radial Attention uses sparsity ratios of 67.7, 76.9, 80.6, 82.4, 83.5, 84.9, and 86.5 across 7 chunks for 5-second videos.The effective sparsity ratio changes with the key-value sequence length over autoregressive chunks.
  • A. Implementation details of Baselines: SVG2 is configured with 50 query centroids, 100 key centroids, 20 initialization iterations, top-p K-means 0.9, minimum key-centroid ratio 0.10, and 2 iteration steps.K-means clustering is re-initialized whenever the key-value length changes in autoregressive models.
  • A. Implementation details of Baselines: Light Forcing provides duration-specific settings for 5-second and 15-second generation, covering target sparsity, retained past frames, historical-frame selections, and model links.The listed parameters include npast keep, nsink, and nwin.
  • A. Implementation details of Baselines: Hierarchical Sparse Attention activates only when historical frames exceed npast keep; otherwise, all historical frames are preserved.This condition is part of the Light Forcing implementation settings.
  • B. Prompts for Long Video Generation: Prompt 1 describes a 3D animation of a blue-furred rabbit-squirrel creature exploring an enchanted forest.The scene includes a sparkling stream, glowing color-changing flowers, purple and silver leaves, and floating lights resembling fireflies.
  • B. Prompts for Long Video Generation: Prompt 2 describes an astronaut in a white spacesuit walking between moss-covered, intricately carved stone buildings.The setting includes a barren landscape, distant rocky hills, and a pale orange sky.

C. Theoretical proof of CAG · D. Detailed VBench Results · E. Detailed VBench-Long Results

The paper theoretically analyzes CAG through a chunk-wise denoising-with-re-noising Markov process, deriving KL and conditional TV controls under mean-map error assumptions. It also reports comprehensive VBench and VBench-Long evaluations, including quality, semantic, latency, and 15-second-generation comparisons.

  • C. Theoretical proof of CAG: CAG models each autoregressive chunk with a denoising-with-re-noising Markov kernel under a fixed inference schedule.The transition operator defines the stochastic update across noise levels, with Gaussian transitions conditioned on the current state.
  • C. Theoretical proof of CAG: The proof compares the generated chunk distribution q0(· | x<i, c) with the true conditional distribution p0(· | x<i, c) using an ideal reverse mean map and model-error assumptions.The analysis assumes conditional mean-map accuracy linked to score or denoiser estimation accuracy.
  • C. Theoretical proof of CAG: Gaussian KL control and Markov-chain KL telescoping yield a conditional total-variation guarantee between generated and true chunk distributions.The argument first bounds per-step Gaussian KL and then applies the KL chain rule along the induced Markov chain.
  • C. Theoretical proof of CAG: The error analysis separates denoising-step discretization or mixing error from model score error, with exact-score sampling error decaying as ˜O(d/T).For imperfect models, the bound instead saturates according to the model-error term.
  • D. Detailed VBench Results: The full VBench evaluation reports performance across all dimensions in separate quality and semantic comparisons against state-of-the-art baselines.These results are provided in Tables 6 and 7.
  • D. Detailed VBench Results: Table 8 compares feature-cache and token-reduction acceleration methods using latency and representative VBench metrics.The table focuses on acceleration-oriented comparisons rather than the full VBench dimensions.
  • E. Detailed VBench-Long Results: The paper reports full VBench-Long results for 15-second video generation in Table 9.This extends evaluation to longer-duration generation using the VBench-Long benchmark.

F. Qualitative Ablation Study

Long-video qualitative ablations show that fixed sparsity causes severe visual degradation, while Chunk-Aware Growth substantially improves color consistency by adapting sparsity to chunk difficulty. The study evaluates the separate contributions of Chunk-Aware Growth and Hierarchical Sparse Attention in the trained long-video model.

  • Ablation setup: The ablation compares Chunk-Aware Growth and Hierarchical Sparse Attention using the trained long-video version of Light Forcing.All variants are evaluated through long-video qualitative ablations.
  • Naive baseline: The naive baseline applies a fixed 85% sparsity ratio across chunks and suffers severe color degradation with noticeable noise from around 5s.The noise appears in the first example of the qualitative comparison.
  • Chunk-Aware Growth: Adding Chunk-Aware Growth largely alleviates color degradation by allocating sparsity according to chunk-wise generation difficulty.White band-like artifacts nevertheless remain in later video portions, such as around 10s.

G. Comparison with Other Acceleration Methods · H. More Efficiency Analysis · H.1. More Devices

Light Forcing is compared with feature caching and token reduction, achieving the best reported accuracy at 1.30× speedup. Across RTX 5090, A100, and H100 platforms, it consistently improves end-to-end latency, with larger gains when combined with the complete deployment stack.

  • G. Comparison with Other Acceleration Methods: Light Forcing is evaluated against feature caching and token reduction as acceleration paradigms beyond sparse attention.The compared methods are TeaCache and DyCoke.
  • G. Comparison with Other Acceleration Methods: TeaCache uses timestep-embedding output changes to decide when intermediate denoising outputs can be reused.
  • G. Comparison with Other Acceleration Methods: DyCoke is adapted to Self Forcing’s chunk-wise generation by grouping three consecutive frames into one unit.The implementation follows LLMC+ and matches Self Forcing’s three-frame chunk protocol.
  • G. Comparison with Other Acceleration Methods: 50% historical KV-cache retention is achieved by keeping all first-frame tokens and 25% of second- and third-frame tokens within each group.
  • G. Comparison with Other Acceleration Methods: At 1.30× speedup, Light Forcing achieves the best accuracy across all reported representative metrics, including subject and background consistency, imaging quality, multiple-object reasoning, and scene understanding.The reported Light Forcing scores are 96.2, 96.5, 71.0, 88.9, and 55.3, respectively.
  • H.1. More Devices: The hardware evaluation uses single-GPU inference on RTX 5090, A100, and H100 platforms after operator warm-up.Timing starts from the second generated sample and uses a common Docker image unless otherwise specified.
  • H.1. More Devices: 3.07×–3.17× speedup is reached on RTX 5090 with the complete deployment stack, while sparse attention alone provides 1.33× and 1.26× acceleration for 5s and 15s generation.
  • H. More Efficiency Analysis · H.1. More Devices: 1.11×–1.12× speedup is provided by Light Forcing alone on H100, with nearly 2× acceleration when combined with other deployment optimizations.The passage notes that attention computation for the 1.3B model is already relatively lightweight on H-series GPUs.

H.2. Peak Memory Analysis · I. More Visualization Examples

The peak-memory analysis measures RTX 5090 usage and finds that sparse attention alone leaves peak memory unchanged, while FP8 layers, efficient kernels, and LightVAE reduce memory overhead. Additional visualizations show that most baselines suffer detail loss and anomalous generations across two prompts.

  • H.2. Peak Memory Analysis: Peak GPU memory is measured on an RTX 5090 under the same Docker image and inference protocol as the preceding experiments.The measurements are reported in Table 11.
  • H.2. Peak Memory Analysis: Table 11 reports peak memory usage on the RTX 5090 in GB.
  • H.2. Peak Memory Analysis: Sparse attention alone leaves measured peak memory unchanged because Light Forcing primarily reduces runtime attention tiles rather than memory allocation.Its main benefit in the current implementation is latency reduction.
  • H.2. Peak Memory Analysis: FP8 linear layers and efficient kernels reduce memory footprint by lowering activation or operator overhead.
  • H.2. Peak Memory Analysis: LightVAE provides the largest memory reduction by replacing the VAE component with a more efficient alternative.
  • I. More Visualization Examples: Figure 10 compares all baselines on the prompts “A person is clay pottery making” and “Turtle swimming in ocean”.The supplementary material provides these detailed qualitative comparisons.
  • I. More Visualization Examples: Most baselines show noticeable degradation through lost fine-grained details, such as distorted hands, and anomalous generations, such as a two-headed turtle.

J. Limitations

The work’s limitations include evaluation only on 1.3B models, residual subtle artifacts in a small number of samples, and less pronounced kernel-fusion acceleration on certain GPUs.

  • Model scale: Evaluation is limited to 1.3B models, leaving scaling Light Forcing to larger models as an important direction.The paper cites a 14B realtime-video model as an example of a larger target model.
  • Artifacts: Subtle artifacts may remain in a very small number of samples despite HSA alleviating white band-like artifacts.The authors leave further artifact reduction as future work.
  • Hardware efficiency: Kernel-fusion operators may provide less pronounced acceleration on certain GPUs.
Loading 2602.04789v4…