PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-free and Adaptive Sparse Attention for Efficient Long Video Generation

Yifei Xia, Suhan Ling, Fangcheng Fu, Yujie Wang, Huixia Li, Xuefeng Xiao, Bin Cui

arXiv:2502.21079v1cs.CV

TL;DR

Long-video DiT generation is bottlenecked by attention cost, and existing sparse methods do not accurately adapt to DiTs’ input-dependent, dispersed sparsity. AdaSpa combines dynamic blockified patterns with online precise search, achieving reported acceleration while preserving video quality and requiring no training or profiling.

  • Problem

    Attention dominates the computational cost of long-video DiT generation, while existing dynamic methods cannot accurately adapt to DiTs’ input-dependent and complex sparse patterns.

  • Method

    AdaSpa uses dynamic blockified sparse patterns with Fused LSE-Cached Online Search and Head-Adaptive Hierarchical Block Sparse attention.

  • Results

    AdaSpa consistently achieves the best quality-efficiency performance across experiments, reaching a highest speedup of 1.78× on HunyuanVideo and 1.66× on CogVideoX1.5-5B.

  • Takeaways & Limitations

    AdaSpa provides a training-free, data-free, plug-and-play approach for accelerating long-video generation while maintaining high generated-video quality.

  • Takeaways & Limitations

    Existing dynamic-pattern methods have limited portability and accuracy on DiTs because sparsity varies with inputs and sparse indices are dispersed.

Abstract

from arXiv · show

Generating high-fidelity long videos with Diffusion Transformers (DiTs) is often hindered by significant latency, primarily due to the computational demands of attention mechanisms. For instance, generating an 8-second 720p video (110K tokens) with HunyuanVideo takes about 600 PFLOPs, with around 500 PFLOPs consumed by attention computations. To address this issue, we propose AdaSpa, the first Dynamic Pattern and Online Precise Search sparse attention method. Firstly, to realize the Dynamic Pattern, we introduce a blockified pattern to efficiently capture the hierarchical sparsity inherent in DiTs. This is based on our observation that sparse characteristics of DiTs exhibit hierarchical and blockified structures between and within different modalities. This blockified approach significantly reduces the complexity of attention computation while maintaining high fidelity in the generated videos. Secondly, to enable Online Precise Search, we propose the Fused LSE-Cached Search with Head-adaptive Hierarchical Block Sparse Attention. This method is motivated by our finding that DiTs' sparse pattern and LSE vary w.r.t. inputs, layers, and heads, but remain invariant across denoising steps. By leveraging this invariance across denoising steps, it adapts to the dynamic nature of DiTs and allows for precise, real-time identification of sparse indices with minimal overhead. AdaSpa is implemented as an adaptive, plug-and-play solution and can be integrated seamlessly with existing DiTs, requiring neither additional fine-tuning nor a dataset-dependent profiling. Extensive experiments validate that AdaSpa delivers substantial acceleration across various models while preserving video quality, establishing itself as a robust and scalable approach to efficient video generation.

1. Introduction

Long-video generation with DiTs is computationally expensive because attention dominates inference cost, while existing sparse-pattern methods struggle with DiTs’ dynamic, irregular sparsity. AdaSpa addresses this gap with blockified dynamic patterns and online precise search, without training or data-dependent profiling.

  • Motivation: 600 PFLOPs are required to generate an 8-second 720p HunyuanVideo clip, with nearly 500 PFLOPs consumed by attention.Attention’s share increases with video resolution or duration.
  • Motivation: Sparse attention reduces computation by omitting redundant token interactions through sparse patterns and retained sparse indices.Existing approaches include static patterns and mixed patterns selected during execution.
  • Limitations of Existing Methods: Static patterns cannot flexibly capture DiTs’ highly dynamic and irregular sparse indices, leading to poor performance.Their fixed structure is mismatched to DiT sparsity.
  • Limitations of Existing Methods: Dynamic-pattern methods also struggle because DiT sparsity varies with inputs and dispersed key regions hinder approximate sparse-index search.Offline search lacks portability and approximation can miss complex sparse indices.
  • AdaSpa: AdaSpa introduces the first Dynamic Pattern plus Online Precise Search method, using a blockified pattern for DiTs’ hierarchical sparsity.The method is designed to accelerate video generation while preserving quality.
  • AdaSpa: AdaSpa combines Fused LSE-Cached Online Search with Head-Adaptive Hierarchical Block Sparse attention in a training-free, data-free framework.Its optimized kernel reduces online search time to under 5% of full-attention generation time, and the plug-and-play handler requires no fine-tuning or data profiling.

2. Preliminaries

DiTs use attention to model spatial, temporal, and cross-modal dependencies, but full attention becomes prohibitively expensive for long sequences. Sparse and blockwise methods reduce computation by omitting low-weight interactions while retaining important attention structure.

  • Diffusion Transformers and 3D Full Attention: DiTs combine video and text tokens in 3D full attention to improve multimodal fusion.Video and text tokens form one unified sequence, with spatial, temporal, and cross-modal dependencies modeled through self-attention.
  • FlashAttention: FlashAttention computes attention blockwise without materializing the full attention matrix, reducing memory to O(Lb).It processes smaller query-key blocks sequentially and uses online_softmax to obtain the final attention output.
  • Sparse Attention: Sparse attention masks interactions with small attention weights, retaining selected key-value pairs to reduce computation.The mask identifies retained token interactions, while removed interactions determine the sparsity level.
  • Sparse Attention: Recall measures how well a sparse pattern preserves dense attention behavior, with higher Recall indicating better retention.Sparse patterns are represented by the retained indices of the masking function.

3. Sparse Pattern Characteristic in DiTs

DiT attention sparsity is hierarchical, locally structured, and dependent on inputs, layers, and heads, making conventional continuous or offline patterns inadequate. The paper therefore motivates blockified patterns and precise online search that reuse invariants across denoising steps.

  • Hierarchical Sparse Structure: DiT attention has a hierarchical organization across text, video, and frame regions, making global continuous sparse patterns ineffective.Video-video attention decomposes into f × f frame regions, while text-related blocks form distinct modality interactions.
  • Hierarchical Sparse Structure: Within frame regions, sparsity can be locally structured, but uneven attention concentration across regions prevents frame-wise search from producing a globally optimized pattern.Nonzero weights concentrate in only a few frame regions rather than being evenly distributed.
  • Blockified Pattern: The blockified pattern captures hierarchical DiT sparsity more effectively than conventional continuous patterns and consistently achieves the best recall.It represents locally structured patterns together with weak or varying inter-region interactions, including sliding-window, random-block, and sink effects.
  • Input and Model Dependence: Sparse patterns vary with inputs, layers, and heads, so patterns searched offline do not reliably transfer across prompts or random seeds.A pattern optimized for one input is not necessarily optimal for other inputs.
  • Online Search: Approximate online search cannot accurately capture dispersed hierarchical sparse indices, while precise online search is computationally prohibitive.This combination motivates a search strategy that is both precise and computationally efficient.
  • Denoising-Step Invariance: For a fixed layer and head, sparse patterns and LSE distributions remain stable across denoising steps, enabling cached reuse for faster precise search.The specific attention weights change across steps, but the underlying sparse pattern remains consistent and LSE distributions remain stable.

4. Methodology

AdaSpa combines blockified sparse attention with online precise search to adapt sparse masks across heads and denoising steps. Its fused LSE-cached search and hierarchical head adaptation reduce attention computation while preserving precise sparse-index selection.

  • Overview: AdaSpa targets long-video DiTs with Dynamic Pattern and Online Precise Search sparse attention.The method is designed to accelerate long-video generation with DiTs.
  • Blockified Sparse Attention: Block sparse attention partitions the sequence into blocks and omits discarded blocks according to selected sparse indices.A sufficiently large negative bias excludes discarded blocks from safe Softmax computation.
  • Blockified Sparse Attention: Optimal block indices retain the largest attention-weight sums, reducing complexity from O(L^2d) to O((1 − sparsity)L^2d).The selected indices are obtained by computing blockwise attention-weight sums and applying topk.
  • Online Precise Search: AdaSpa uses fused LSE-cached search because precise online mask construction otherwise requires costly O(L^2) attention weights at each operation.The fused search computes FlashAttention outputs and row LSE values, then reuses LSE to compute blockwise attention-weight sums.
  • Head-adaptive Attention: Head-adaptive hierarchical search assigns different sparsity levels to heads according to Recall while preserving average sparsity.Higher-Recall heads receive more sparsity, whereas lower-Recall heads receive less, improving precision without changing average sparsity.
  • Adaptive Scheduling: AdaSpa performs full attention during warmup, searches selected steps, and reuses the resulting masks across subsequent denoising steps.Its implementation is exposed as a plug-and-play interface with one-line activation and configurable sparsity, block size, and search steps.

5. Experiments

Experiments compare AdaSpa with static and dynamic baselines across two video-generation models using quality, similarity, and efficiency metrics. AdaSpa achieves the strongest overall quality-efficiency results, remains robust as sparsity and warmup settings vary, and scales to longer videos.

  • Main Results: AdaSpa achieves the best quality and efficiency across experiments, reaching 1.78× speedup on HunyuanVideo and 1.66× on CogVideoX1.5-5B.The evaluation covers HunyuanVideo and CogVideoX1.5-5B at long-video settings with established video-quality and similarity metrics.
  • Main Results: AdaSpa outperforms MInference and Sparse VideoGen because it more accurately captures dynamic sparse indices across heads.MInference’s approximate search has the lowest accuracy, while Sparse VideoGen’s static pattern cannot adapt accurately to all heads.
  • Quality–Sparsity Trade-off: AdaSpa maintains the highest VBench quality across sparsity levels without significant degradation as sparsity increases.Sparse VideoGen and MInference experience considerable quality drops under increasing sparsity.
  • Quality–Sparsity Trade-off: As sparsity increases, AdaSpa retains higher PSNR, SSIM, and LPIPS similarity with a gradual decline rather than MInference’s abrupt drop.These metrics compare sparse-attention outputs with outputs generated without sparse attention.
  • Warmup: Increasing warmup decreases similarity when reduced, while AdaSpa remains best across warmup setups and video quality stays nearly unchanged.The results indicate that warmup primarily affects similarity to the original video rather than video quality itself.
  • Search Strategy: Additional searches improve accuracy only slightly and can eventually reduce video quality, supporting similarity of sparse patterns across denoising steps.This result motivates limiting search frequency rather than searching every step.
  • Scalability: AdaSpa’s speedup increases with video length, reaching 4.01× for 24-second videos under the tested configuration.The scaling test uses sparsity=0.9, block_size=64, and Ts = {0, 30}.

6. Conclusion

AdaSpa is developed from analysis of DiT attention sparsity to accelerate long video generation through dynamic patterns and online precise search. It achieves 1.78× efficiency improvement while maintaining high video quality.

  • AdaSpa combines dynamic pattern selection with online precise search to accelerate long video generation.The approach is presented as a sparse attention method developed from analyses of DiT attention mechanisms.
  • The method targets efficient video generation through a new sparse attention approach based on observed DiT sparsity characteristics.
  • 1.78× efficiency improvement is achieved while maintaining high quality in generated videos.
Loading 2502.21079v1…