Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search

1. Introduction

• 3D full attention over unified video and text tokens makes inference costly as resolution and temporal length increase; a 125-frame 720p HunyuanVideo generation takes about 38 minutes.On one NVIDIA A800, self-attention accounts for roughly 80% of total latency and processes more than 115K tokens.Source: §1 · p. 1
• Existing sparse attention methods use fixed sparsity ratios or thresholds and face difficulty accurately and efficiently identifying critical blocks.Static patterns can misalign with ground truth, while runtime importance evaluation can be expensive or less accurate.Source: §1 · p. 1
• Attention entropy and mean attention distance generally decrease over denoising, motivating more computation early and progressively greater sparsity later.Lower entropy indicates focus on fewer tokens, and shorter distance indicates more localized attention.Source: §1 · p. 1
• Blocks selected early continue to capture a large fraction of later attention weight, suggesting subsequent searches can reuse that candidate set.Restricting later searches to early candidates reduces search overhead while maintaining high coverage of critical blocks.Source: §1 · p. 2
• GSRS combines growing sparsity and reduced search with Top-p block selection and precise block-importance evaluation fused into FlashAttention at negligible overhead.It retains more computation early, progressively lowers selection thresholds, and restricts later searches to an early candidate set.Source: §1 · p. 2
• Up to 2.21× speedup with generation quality comparable to full attention is reported for GSRS on HunyuanVideo and Wan2.1.The paper also reports a superior quality-efficiency trade-off compared with evaluated sparse attention methods.Source: §1 · p. 2

Sparse Attention in LLMs

• MInference [19], MoBA [32], and NSA [59] explore sparse attention designs, but are designed for single-forward inference and do not account for DiT multi-step iterative denoising, limiting direct applicability to video generation.Source: §2 · p. 2

Sparse Attention in Video DiTs

• Video-DiT sparse methods use static patterns or runtime block selection, but do not exploit changing attention behavior across denoising steps.Static approaches offer simplicity but limited adaptability; dynamic approaches face a balance between selection accuracy and search overhead.Source: §2 · p. 2

Other Acceleration Techniques for DiTs

• Caching, low-bit quantization, step distillation, and variable-resolution sampling are presented as complementary ways to accelerate DiTs.The passage states these techniques can be combined with sparse attention for further speedup.Source: §2 · p. 2

3.1. Attention and FlashAttention

• Self-attention projects tokens into query, key, and value matrices, while FlashAttention computes attention incrementally without materializing the full L × L matrix.FlashAttention maintains a per-row Log-Sum-Exp normalization factor that is later used for precise block-importance evaluation.Source: §3.1 · p. 2
• Figure 1 tracks attention entropy and mean attention distance across 10 prompts, with different shades showing individual samples.Tokens are reordered via Hilbert curves to preserve spatial-temporal adjacency before positional distances are computed.Source: Figure 1 · p. 3

3.2. Block Sparse Attention

• Block-sparse attention skips unselected blocks during the FlashAttention tiling loop, avoiding their computation and memory access.Attention recall measures the fraction of total attention weight captured by selected blocks; Figure 2 reports high recall for an early mask held fixed at later steps on both models.Source: §3.2 · p. 3, Figure 2 · p. 3

Attention dynamics

• Attention entropy and mean attention distance generally decrease across denoising steps on HunyuanVideo and Wan2.1, indicating attention focuses on fewer and closer tokens.The reported trend is consistent across architectures and independent of specific inputs, motivating increased sparsity later in denoising.Source: §4 · p. 3
• Blocks with high attention weights early in denoising continue to capture substantial weight later, while blocks with low early weights tend to remain low.An early candidate set retains a large fraction of total attention weight at later steps on both HunyuanVideo and Wan2.1, motivating restricted subsequent searches.Source: §4 · pp. 3–4, Figure 3 · p. 4

5.1. Growing Sparsity and Reduced Search

• GSRS progressively lowers the Top-p threshold across search steps, retaining more blocks early and allowing greater sparsity later as attention becomes more concentrated.The decreasing threshold schedule outperforms fixed and shuffled p configurations under comparable computational budgets.Source: §5.1 · p. 4
• After full attention warmup and an initial full-space search, GSRS searches within the baseline candidate set, reuses the latest mask between searches, and can reselect previously omitted blocks within that set.The early candidate set continues to capture a large fraction of later attention weight.Source: §5.1 · pp. 4–5, Figure 4 · p. 5

5.2. Adaptive Block Selection via Top-p.

• Top-p retains blocks in descending importance order until their cumulative importance exceeds p, adapting the number retained to each head’s attention distribution without manual tuning.This contrasts with fixed Top-k, which assigns every head the same number of blocks.Source: §5.2 · p. 5

5.3. Precise Block Importance Evaluation

• Block importance is the average attention weight a query block assigns to a key block, and sums to 1 across key blocks for each query block, enabling Top-p selection.FlashAttention produces the global row-wise normalization factor only after processing the last key block, so exact weights are not available during the computation.Source: §5.3 · pp. 5–6
• Cached row-wise LSE values can drift as attention changes across denoising steps, making block-importance sums inaccurate and causing Top-p to discard important blocks.GSRS avoids cached LSE values when recovering block importance.Source: §5.3 · p. 6
• GSRS records per-block log-sum-exp statistics during FlashAttention, then normalizes them with the current row-wise LSE to recover exact block importance with negligible latency and memory overhead.The two phases require no additional pass over the data.Source: §5.3 · p. 6

6.1. Experimental Setup

• Experiments use Wan2.1-T2V-14B and HunyuanVideo-T2V-13B on VBench, with 720p generation, 50 denoising steps, and a single NVIDIA A800 GPU.The default configuration uses full-attention warmup for the first 20% of denoising steps, then searches at T = {10, 20, 30, 40} with thresholds p = {0.95, 0.9, 0.85, 0.8}.Source: §6.1 · pp. 6–7

6.2. Main Results

• GSRS records PSNR scores of 32.75 on HunyuanVideo and 23.79 on Wan 2.1, outperforming all baselines; its VBench-Q scores are 84.01% and 83.72%, respectively.GSRS has the highest PSNR and SSIM on both models, and the highest VBench-Q among evaluated sparse methods.Source: §6.2 · p. 7, Table 1 · p. 7
• 1.55× and 1.68× speedups on Wan 2.1 and HunyuanVideo accompany densities of 30.63% and 25.06%, respectively; H800 speedups reach 1.78× and 2.21%.GSRS achieves the highest speedup among evaluated sparse methods on both models; its VBench-Q scores are comparable to full attention.Source: §6.2 · p. 7, Table 1 · p. 7

6.3. Ablation Study

• The decreasing Top-p schedule improves PSNR from 31.20 to 31.96 and reduces latency from 1852s to 1825s versus fixed p = 0.93.The shuffled schedule reaches PSNR 31.64 at similar latency; the decreasing schedule also has the highest PSNR and VBench-Q among compared configurations.Source: §6.3 · p. 8, Table 2 · p. 8
• Removing Reduced Search raises density from 25.06% to 28.62% and latency from 1825s to 1990s, while reduced search cuts search overhead 13.75×.Previous-mask search lowers latency to 1757s but yields PSNR 30.87 and VBench-Q 84.80, supporting dynamic selection within the baseline mask.Source: §6.3 · p. 8, Table 3 · p. 8
• Replacing precise block-importance evaluation with cached LSE values lowers PSNR from 31.96 to 25.32 and VBench-Q from 84.99 to 84.04.The passage attributes this to distorted cumulative probability sums and block rankings, which make selection inaccurate.Source: §6.3 · p. 8, Table 3 · p. 8
• At matched density, Top-k (25%) achieves PSNR 29.84 versus Top-p’s 31.96 at similar latency; Top-k (50%) takes 1304s for only marginal VBench-Q improvement.The ablation concludes adaptive Top-p selection allocates computation more effectively than fixed budgets.Source: §6.3 · p. 8, Table 5 · p. 8
• Among tested search-step configurations, {10, 20, 30, 40} has the highest VBench-Q while maintaining low latency; all achieve PSNR above 31.89.The four-step configuration provides the best overall quality-efficiency trade-off in this ablation.Source: §6.3 · p. 8, Table 4 · p. 8

• GSRS combines growing sparsity and reduced search in a training-free sparse-attention framework, achieving up to 2.21× speedup with generation quality comparable to full attention.Source: §7 · p. 8

A.1. FlashAttention Details

• Online softmax accumulates the numerator and denominator block by block, postponing final division because the full score vector is not held in memory.When a new maximum appears, previously accumulated results are rescaled; the true LSE is available only after the final block.Source: §A.1 · pp. 12–13
• The two-phase correction uses cached LSE as an in-kernel approximation, then corrects block weights with the true LSE after the kernel completes.This addresses the incomplete normalization available during online softmax.Source: §A.1 · p. 13

A.2. Equivalence to the Block Importance Definition

• After candidate blocks are processed, the recovered weights equal block importance normalized over evaluated key tokens, so their sum is one for Top-p selection.The evaluated set contains all key blocks at the first search and the early candidate set at subsequent searches.Source: §A.2 · p. 14

A.3. Fused Precise Block Importance Evaluation

• Fused block-importance evaluation executes Phase 1 within the FlashAttention kernel and a lightweight post-kernel Phase 2; GSRS achieves slightly lower peak memory than full attention despite additional storage for ˆSiv.The comparison is during inference on HunyuanVideo using a single A800 for 125 frames at 720p.Source: §A.3 · p. 15, Table 6 · p. 15

A.4. Top-p Block Selection

• Top-p selection sorts block-importance weights and selects the shortest descending prefix whose cumulative sum reaches the threshold.This makes the selected block count depend on the distribution of importance weights.Source: §A.4 · p. 16

A.5. GSRS Pipeline

• GSRS begins with full-attention warmup, performs a full-space first search, and re-evaluates blocks in the early candidate set at later search steps.Between searches, it uses the latest selected mask; the current LSE is not cached across denoising steps.Source: §A.5 · p. 17

A.6. Token Reordering via Hilbert Curve

• Generalized Hilbert-curve reordering groups nearby spatial-temporal tokens into compact blocks, helping block-sparse attention capture local interactions.The paper applies this reordering in all experiments.Source: §A.6 · p. 17, Figure 6 · p. 17

A.7. GSRS Runtime Breakdown

• 13.75× lower total overhead: reduced search cuts the extra attention and search overhead measured against GSRS without reduced search on HunyuanVideo.The comparison attributes overhead to evaluating all blocks, which increases both search time and the number of retained blocks.Source: Figure 7 · p. 18, §A.7 · p. 18

C. Effect of Warmup Length

• Increasing warmup steps improves similarity to full attention at the cost of higher latency; the authors choose Nw = 10 to balance similarity and speedup.VBench-Q remains between 83.71% and 84.01% across warmup settings, and is reported as largely preserved even with Nw = 1.Source: C. Effect of Warmup Length · p. 19, Table 8 · p. 19, Table 9 · p. 19

D. PSNR Distribution

• GSRS has higher median PSNR than baselines on Wan2.1 and HunyuanVideo, with higher lower and upper quartiles across VBench prompts.The distribution is described as consistent with average scores in Table 1.Source: D. PSNR Distribution · p. 19, Figure 8 · p. 19

E. Performance Across Random Seeds

• The five-seed HunyuanVideo evaluation reports mean and standard deviation for PSNR, SSIM, LPIPS, and VBench-Q; similarity metrics use same-seed full-attention outputs.Table 10 is identified as the five-seed performance table.Source: E. Performance Across Random Seeds · p. 20, Table 10 · p. 20

F. Quality-Efficiency Trade-off

• The Top-p configuration study evaluates quality-efficiency using PSNR, SSIM, LPIPS, VBench-Q, inference latency, and speedup.The supplied passage identifies the compared measures but does not provide their values.Source: F. Quality-Efficiency Trade-off · p. 20, Table 11 · p. 20

G. Inference Speed on NVIDIA H800

• The H800 comparison reports inference latency and speedup relative to full attention on the same GPU.Table 12’s caption identifies the H800 setting; no comparison values are supplied here.Source: G. Inference Speed on NVIDIA H800 · p. 20, Table 12 · p. 20

H. Limitations

• GSRS requires at least one full-attention warmup step for the initial LSE, and its generalization beyond DiT-based video generation remains an open question.The authors state that removing warmup could further improve efficiency.Source: H. Limitations · p. 21

I. Broader Impacts

• The paper notes that faster, cheaper generation may lower barriers to both access and misuse, including large-scale deceptive video production.It recommends safeguards such as watermarking and synthetic-media detection.Source: §I · p. 21

J. Detailed Comparison with AdaSpa and LiteAttention

• GSRS re-evaluates all blocks in the early candidate set at later searches, allowing previously omitted blocks there to be selected again; LiteAttention excludes skipped blocks thereafter.The paper describes this as retaining dynamic selection while reducing search overhead compared with evaluating all blocks at each search step.Source: J. Detailed Comparison with AdaSpa and… · p. 21
• GSRS uses block log-sum-exp statistics and current row-wise LSE for normalized importance, whereas AdaSpa uses cached LSE and LiteAttention uses a local skipping condition.The paper notes that cached versus current-step normalization affects cumulative probability thresholds and block rankings.Source: J. Detailed Comparison with AdaSpa and… · pp. 21–22
• Under a matched 20% keep ratio, GSRS improves PSNR and SSIM over AdaSpa without head-adaptive allocation and outperforms the Ampere LiteAttention implementation in PSNR, SSIM, and speedup.The authors state that the gains are not solely attributable to Top-p selection or head-adaptive sparsity allocation.Source: J. Detailed Comparison with AdaSpa and… · p. 22, Table 13 · p. 22

K. More Details about Attention Entropy and Mean Attention Distance

• Entropy and mean attention distance are accumulated within FlashAttention kernels, avoiding the O(L^2) memory cost of materializing full attention maps.The entropy calculation uses a running weighted-score sum; the fused diagnostics add negligible computation and no additional memory beyond storing the resulting metrics.Source: K. More Details about Attention… · p. 23, §K.1 · p. 23, §K.2 · p. 23

K.3. Attention Dynamics Across Prompts

• Across diverse prompts for HunyuanVideo and Wan2.1, entropy and mean attention distance decrease monotonically during denoising, with only minor variation in absolute values.The authors characterize this pattern as intrinsic to the model architecture rather than content-dependent.Source: §K.3 · p. 23, Figure 9 · p. 24

K.4. Attention Dynamics Across Layers

• Across layers, both metrics decrease during denoising, but persistent layer variation means some layers remain more dispersed while others stay concentrated and localized.For Wan2.1, conditional and unconditional branches show similar layer-wise patterns; the authors say these observations support layer-adaptive sparsity.Source: §K.4 · p. 25, Figure 10 · p. 25

K.5. Attention Dynamics Across Heads

• Heads within a layer differ in attention concentration, while entropy and mean attention distance decrease across denoising steps.The head-level visualizations cover HunyuanVideo and both Wan 2.1 branches.Source: §K.5 · p. 25, Figure 11 · p. 26, Figure 12 · p. 27
• Wan 2.1's conditional and unconditional branches show similar head-wise patterns, suggesting head differences are not caused by conditioning.Source: §K.5 · p. 26, Figure 13 · p. 28

L. Visualization of the generated videos

• GSRS preserves high pixel-level fidelity and achieves generation quality similar to full attention on Wan 2.1 and HunyuanVideo.The visual comparisons are shown in Figures 14 and 15.Source: L. Visualization of the generated videos · p. 29, Figure 14 · p. 29, Figure 15 · p. 30