PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation

Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Jianfei Chen, Song Han, Kurt Keutzer, Ion Stoica

arXiv:2505.18875v5cs.CV

TL;DR

DiT-based video generation is bottlenecked by quadratic attention cost, while existing sparse attention methods lose quality through inaccurate token identification and scattered computation. SVG2 addresses both issues with semantic-aware permutation and achieves a superior quality–efficiency trade-off, including speedups up to 2.30× while maintaining PSNR up to 30 on HunyuanVideo.

  • Problem

    Existing sparse attention methods inaccurately identify critical tokens through position-based clustering and waste computation because critical tokens are scattered among non-critical ones.

  • Method

    SVG2 is a training-free sparse attention approach that clusters and reorders tokens by semantic similarity to improve critical-token identification and create contiguous sparse layouts.

  • Results

    SVG2 achieves a superior generation-quality and efficiency trade-off, with end-to-end speedups up to 2.30× and PSNR up to 30 on HunyuanVideo while outperforming prior methods.

  • Takeaways & Limitations

    Semantic-aware permutation improves attention recall and reduces computation overhead by an average of 36% without changing the selected critical tokens.

  • Takeaways & Limitations

    SVG2’s independent clustering produces cross-category centroids and a many-to-many map in which multiple query clusters can share important key clusters.

Abstract

from arXiv · show

Diffusion Transformers (DiTs) are essential for video generation but suffer from significant latency due to the quadratic complexity of attention. By computing only critical tokens, sparse attention reduces computational costs and offers a promising acceleration approach. However, we identify that existing methods fail to approach optimal generation quality under the same computation budget for two reasons: (1) Inaccurate critical token identification: current methods cluster tokens based on position rather than semantics, leading to imprecise aggregated representations. (2) Excessive computation waste: critical tokens are scattered among non-critical ones, leading to wasted computation on GPUs, which are optimized for processing contiguous tokens. In this paper, we propose SVG2, a training-free framework that maximizes identification accuracy and minimizes computation waste, achieving a Pareto frontier trade-off between generation quality and efficiency. The core of SVG2 is semantic-aware permutation, which clusters and reorders tokens based on semantic similarity using k-means. This approach ensures both a precise cluster representation, improving identification accuracy, and a densified layout of critical tokens, enabling efficient computation without padding. Additionally, SVG2 integrates top-p dynamic budget control and customized kernel implementations, achieving up to 2.30x and 1.89x speedup while maintaining a PSNR of up to 30 and 26 on HunyuanVideo and Wan 2.1, respectively. Our code is open-sourced at \href{https://github.com/svg-project/Sparse-VideoGen}{https://github.com/svg-project/Sparse-VideoGen}.

1 Introduction

DiT video generation is bottlenecked by quadratic attention, while existing sparse methods lose quality through inaccurate token identification and hardware-induced computation waste. SVG2 addresses both issues with semantic-aware permutation and achieves a Pareto frontier of quality and efficiency.

  • Generating a five-second HunyuanVideo clip on an NVIDIA A100 takes nearly an hour, with 3D attention exceeding 80% of end-to-end runtime.
  • Existing sparse attention methods fall below oracle generation quality at the same computational budget because of two identified challenges.
  • Position-based blocks mix semantically different tokens, producing unrepresentative aggregated activations and inaccurate critical-token identification.
  • Hardware constraints force computation on entire blocks when only some tokens are critical, causing substantial waste.
  • SVG2 clusters tokens by semantic similarity and permutes them into contiguous blocks, improving identification accuracy while reducing wasted computation.
  • 2.30× and 1.84× end-to-end speedups are achieved on HunyuanVideo and Wan2.1-I2V while maintaining PSNR values up to 30 and 26, respectively.

2 Related Work

Prior work accelerates video generation through static or dynamic sparse attention, alternative attention architectures, and caching or token-compression strategies. These approaches target different efficiency bottlenecks and operating regimes.

  • Static sparse-attention methods predefine patterns offline, while dynamic methods select critical tokens during runtime.
  • Memory-efficient LLM sparse-attention methods reduce memory load but are ineffective for compute-bound DiT video generation.
  • Linear-attention and state-space approaches combine global or local sequence modeling to address long-context latency in video generation.
  • Long-video systems use distillation, KV-cache refreshing, or frame-level token compression to improve efficiency and support longer generation.

3 Motivation

DiT attention is expensive but intrinsically sparse, creating a quality-efficiency opportunity. Existing practical methods approximate attention with coarse position-based blocks, yet lose recall and waste computation because critical tokens are semantically scattered.

  • 3.1 Attention in DiTs is Inherently Sparse: DiTs transform activations into Query, Key, and Value tensors before self-attention produces the output.
  • 3.1 Attention in DiTs is Inherently Sparse: Attention-score computation has quadratic complexity in sequence length and can consume over 80% of end-to-end video-generation time.
  • 3.1 Attention in DiTs is Inherently Sparse: A few largest Q × K^T values dominate softmax attention scores and the resulting weighted output, making attention highly sparse.
  • 3.1 Attention in DiTs is Inherently Sparse: 13% of computations can achieve 95% attention recall, PSNR 27, and up to 2× theoretical end-to-end speedup under oracle token selection.
  • 3.2 Existing Sparse Attention Fails to Match the Oracle Policy: Practical methods cluster consecutive tokens into blocks to approximate attention scores while reducing identification overhead to less than 1% with block size 128.
  • 3.2 Existing Sparse Attention Fails to Match the Oracle Policy: Existing methods trail oracle recall because position-based clusters mix tokens with different latent activations, degrading block representations and critical-token identification.
  • 3.2 Existing Sparse Attention Fails to Match the Oracle Policy: Scattered critical tokens force dense hardware blocks to process non-critical tokens, preventing theoretical sparse-computation savings.

4 Methodology

SVG2 is a training-free sparse-attention framework that uses semantic-aware permutation to improve critical-token identification and reduce computation waste. It combines semantic clustering, centroid-based top-p selection, cached k-means, and dynamic block-sparse kernels for practical video-generation acceleration.

  • Overview: SVG2 uses semantic-aware permutation to target a Pareto frontier between video-generation quality and computational efficiency.The framework dynamically adjusts computation through centroid-based top-p selection.
  • Semantic-Aware Permutation: K-means independently clusters query and key activations so tokens within each cluster share similar semantics and have more precise centroid representations.SVG2 applies clustering separately for each attention head and transformer layer, producing Cq query clusters and Ck key clusters.
  • Semantic-Aware Permutation: Permuting semantically similar tokens into contiguous clusters densifies sparse computation and reduces waste from physically scattered critical tokens.The same key permutation is applied to keys and values to preserve attention-output equivalence.
  • Criticality Estimation: Centroid-based estimation computes approximate attention scores from cluster centroids and cluster sizes to identify critical clusters efficiently.Because the number of clusters is typically below 1024, this approximation usually costs less than 1% of full attention computation.
  • Dynamic Budget Control: Top-p selection sorts clusters by approximate attention score and selects them until a predefined cumulative target is reached.This dynamically determines the number of selected critical tokens rather than using a fixed count.
  • System-Algorithm Co-design: Centroid caching reduces repeated k-means overhead, while dynamic block-sparse kernels support the varied cluster sizes created by permutation.The customized kernel loads contiguous query-cluster tokens and uses per-token address offsets for scattered key/value tokens.

5 Experiment

SVG2 is evaluated on Wan2.1 and HunyuanVideo using video-quality, attention-efficiency, and end-to-end metrics. Across visualizations, ablations, and benchmarks, it improves quality-efficiency trade-offs while reducing computation waste and maintaining high similarity to full attention.

  • Setup: The experiments use 720p Wan2.1 and HunyuanVideo video-generation models, with VBench prompts and image-video pairs.Wan2.1 generates 21 frames and HunyuanVideo 33 frames, each with 3600 tokens per frame after 3D-VAE tokenization.
  • Setup: Quality is measured with PSNR, LPIPS, SSIM, and VBench, while density and FLOPs quantify sparse and end-to-end computation.Density is defined as sparse-attention computation divided by full-attention computation.
  • Setup: SVG2 is compared with Sparse VideoGen, SpargeAttention, and XAttention using official configurations, excluding XAttention on Wan2.1 because it is unsupported.Sparse attention is skipped during the first 30% of denoising steps for all methods.
  • Qualitative Evaluation: Semantic-aware permutation produces contiguous critical-token layouts and recovered attention maps with high similarity to the original maps.The visualization covers diverse sparse patterns across attention heads in Wan2.1 on VBench prompts.
  • Quality Evaluation: 26.5 average PSNR on Wan2.1 and 30.4 on HunyuanVideo: SVG2 outperforms all baselines on PSNR, SSIM, and LPIPS while maintaining the highest speedup.These results are reported in the quality and efficiency comparison.
  • Efficiency Evaluation: 90% attention recall: centroid caching improves k-means efficiency by reducing the density required across different iteration counts.The experiment uses k-means++ and treats lower density as better k-means quality.
  • Efficiency Evaluation: 1.48× average computation reduction and 1.88× reduction in computation waste: customized kernels outperform FlashInfer across dynamic block-size workloads.The practical configuration uses Cq = 100 and Ck = 500 at fixed 90% attention recall.
  • Efficiency Evaluation: 1.82× average speedup: SVG2 maintains the highest generation quality, while SVG2-Turbo achieves 2.5× smaller density than SVG and PSNR of 23.7.SVG2-Turbo demonstrates additional efficiency potential while maintaining quality similar to baselines.

6 Conclusion & Limitation

SVG2 accelerates DiT-based video generation through semantic token clustering and contiguous permutation, improving the quality-efficiency trade-off. The paper’s main limitation is that extension beyond DiT attention remains unevaluated.

  • SVG2 is a training-free sparse-attention approach for accelerating DiT-based video generation.
  • Semantic clustering improves critical-token identification, while contiguous permutation reduces computation waste.
  • The paper does not discuss or evaluate whether SVG2 extends to attention mechanisms beyond DiTs.

A Visualization of the Generated Videos

Visualization comparisons on HunyuanVideo and Wan 2.1 show that SVG2 preserves high pixel-level fidelity relative to dense attention.

  • SVG2 preserves high pixel-level fidelity compared with dense attention on HunyuanVideo and Wan 2.1.The paper states that the generated videos have similar quality to dense attention.
  • Figure 9 compares dense attention and SVG2 for HunyuanVideo and Wan 2.1 text-to-video generation.
  • Figure 10 compares dense attention and SVG2 for Wan 2.1 image-to-video generation.

B Performance Comparison in Warmup-free Setting

In the warmup-free setting, SVG2 consistently provides better quality than the baseline.

  • SVG2 consistently offers better quality than the baseline without warmup steps.
  • The comparison evaluates SVG2 and the baseline with warmup steps set to 0%.

C VBench Results

SVG2 outperforms the other baselines on the reported VBench results, with results provided for both 0% and 30% warmup settings.

  • SVG2 outperforms all other baselines in the full VBench results.
  • Table 3 reports SVG2’s VBench results with warmup steps set to 0%.
  • Table 4 reports SVG2’s VBench results with warmup steps set to 30%.
  • Figures 11 and 12 evaluate the attention kernel by varying query and key cluster counts separately.Figure 11 fixes query clusters while varying key clusters; Figure 12 fixes key clusters while varying query clusters.

D Ablation on the Number of Clusters

The cluster-count ablation shows a quality–efficiency trade-off: more clusters can improve quality, but excessive counts underutilize tensor cores. Q = 100 and K = 500 provide the best reported balance.

  • Cluster-count selection: Q = 100 and K = 500 provide the best balance between generation quality and efficiency.The comparison varies query and key cluster counts while measuring PSNR and end-to-end efficiency.
  • Quality–efficiency trade-off: Increasing the number of clusters generally improves generation quality but can reduce efficiency because of hardware layout constraints.The reported trade-off reflects the interaction between cluster granularity and GPU computation shapes.
  • Hardware constraint: Clusters must contain at least 64 tokens on average to fully utilize tensor cores under the m64n64k16 configuration.This fixed input-size requirement constrains practical query-cluster counts.
  • Hardware constraint: Counts beyond Q = 100 or K = 500 reduce efficiency through underutilization despite potential quality gains.The ablation therefore favors moderate cluster counts rather than maximizing the number of clusters.

D.3 Ablation on Permutation

The permutation ablation finds that Q and K should be clustered independently because their permutation patterns differ substantially. Quality and speedup also vary across models, partly reflecting model-specific numerical sensitivity and attention cost.

  • Permutation comparison: Applying the same Q, K, or shared-QK permutation to both sides produces worse PSNR even with higher density.The evaluated variants reuse one clustering strategy across query and key representations.
  • Permutation comparison: The average Adjusted Rand Index between Q and K clusters is 0.345, supporting independent clustering to preserve attention expressiveness.The low average alignment indicates substantially different Q and K permutation patterns.
  • Model-specific behavior: Wan 2.1 generally reports lower quality than HunyuanVideo because it is more sensitive to numerical precision and backend changes.The passage reports Wan 2.1 PSNR as low as 27–28 across dense-attention backends, versus 33–34 for HunyuanVideo without setup changes.
  • Model-specific behavior: HunyuanVideo reaches 2.30× speedup versus 1.89× for Wan 2.1 because attention contributes differently to their total runtimes.Their context lengths and layer architectures produce different attention-cost ratios, while SVG2 primarily accelerates attention.
Loading 2505.18875v5…