PaperlayerView exact revision on arXiv

Source-linked AI summary

iCATS: Fast Video Generation via Interaction-Aware Sparse Attention and Timestep-Adaptive Sparsity

Chengfeng Han, Baole Ai, Xianlu Bian, Jie Yao, Zilong Huang, Ang Wang, Dandan Ding

arXiv:2610.11302v1cs.CV

TL;DR

Video DiTs face sharply rising attention costs as token sequences grow, while sparse attention can introduce approximation errors that degrade generation quality. iCATS combines interaction-aware importance estimation, SNR-guided sparsity, and tail merging; experiments report a state-of-the-art efficiency-quality trade-off among training-free methods.

  • Problem

    As video sequences grow, DiT attention's quadratic computational cost becomes an inference bottleneck, while sparse attention can introduce quality-degrading approximation errors.

    Source: §1 · p. 2

  • Method

    iCATS is a training-free framework that improves query-key interaction-based importance estimation, adapts sparsity across timesteps using SNR, and reduces execution padding with tail merging.

    Source: §1 · p. 3, §3.1 · p. 4, §4.2 · p. 6

  • Results

    Experiments on representative DiT-based video-generation models show iCATS achieves a state-of-the-art efficiency-quality trade-off among training-free sparse-attention methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    iCATS optimizes importance estimation, sparse-mask construction, and hardware execution within one training-free sparse-attention pipeline.

    Source: §6 · p. 10

    The clustering procedure is not optimized for multi-GPU parallel execution, and the token-level sparse-masking kernel remains to be further optimized.

    Source: §6 · p. 10

Abstract

from arXiv · show

Training-free sparse attention offers a practical acceleration solution to Diffusion Transformers (DiTs) via reducing computations without fine-tuning. It typically involves estimating the importance of query-key regions and deriving sparse masks to compute only the important candidates, which inevitably introduces approximation errors that may degrade generation quality. To better balance the efficiency-quality trade-off, we propose iCATS, integrating improved importance estimation and sparse mask construction with an efficient hardware execution strategy. Specifically, for importance estimation, unlike previous works that perform independent clustering over query and key tokens based on feature similarity to estimate attention scores, iCATS demonstrates that clustering based on query-key dot-product interactions is more accurate and further reformulates this objective as a simple quadratic form for low-cost computation. For sparse mask construction, instead of using a fixed top-p rule, we observe that tolerance to sparse approximation errors varies across denoising timesteps and therefore introduce an SNR-guided sparsity schedule to adjust sparsity dynamically, leading to higher accuracy. Finally, for hardware execution, we devise a tail-merging strategy to reduce padding overhead caused by irregular cluster sizes, improving GPU kernel utilization. Extensive experiments show that iCATS achieves $2.03\times$ acceleration with 31.017 dB PSNR on HunyuanVideo-T2V-13B and $1.55\times$ acceleration with 29.301 dB PSNR on Wan2.1-T2V-14B, delivering a state-of-the-art efficiency-quality trade-off.

1 Introduction

Video DiTs face sharply rising attention costs as sequences grow, motivating inference acceleration that preserves quality. iCATS optimizes importance estimation, timestep-specific sparsity, and GPU execution in a training-free framework.

1 Introduction

  • Growing video-token sequences make attention's quadratic computational cost a critical inference bottleneck for DiT-based generation.The demand for longer and higher-resolution videos increases sequence length.Source: §1 · p. 2
  • Training-free sparse attention adds sparsity without changing model parameters, offering flexibility and plug-and-play use across pretrained DiTs.The introduction contrasts this with training-based methods that require retraining when sparsity changes.Source: §1 · p. 2
  • Query-key interaction clustering reformulates importance estimation as a quadratic-form objective, avoiding full query-key matrix construction while improving estimation accuracy.The objective is defined in the original query/key space.Source: §1 · pp. 2–3
  • An SNR-guided schedule dynamically adjusts the top-p value across denoising timesteps instead of applying a fixed sparsity rule.The top-p value is computed offline from a smooth function of logarithmic SNR.Source: §1 · pp. 2–3
  • Tail merging reduces padding overhead from irregular query-cluster sizes and improves GPU kernel utilization during sparse attention execution.Cluster-based kernels can otherwise repeatedly incur padding overhead on the query side.Source: §1 · pp. 2–3
  • iCATS combines improved importance estimation, adaptive sparse-mask construction, and tail merging to deliver a state-of-the-art efficiency-quality trade-off among training-free methods.The framework is designed to optimize the sparse attention pipeline without modifying model parameters.Source: §1 · p. 3

2 Related Work

Related work spans training-based and training-free sparse attention. Training-free methods avoid parameter updates, but fixed patterns and independent query/key grouping limit adaptability or estimation accuracy.

2.1 Training-based Sparse Attention

  • Training-based sparse-attention methods use learned predictors, coarse-to-fine selection, or multi-level computation, but require model-specific training or fine-tuning.The cited approaches include DSV, VSA, VMoBA, SLA, and SLA2.Source: §2.1 · p. 3

2.2 Training-free Sparse Attention

  • Training-free methods use predefined spatial-temporal patterns or dynamically select regions from token features, avoiding parameter modification.SVG, RainFusion, Sparse-vDiT, and STA use predefined patterns; SpargeAttn, XAttention, and SVG2 construct masks dynamically.Source: §2.2 · p. 3
  • Fixed or hand-designed patterns limit adaptation to diverse video content, while independent query/key feature grouping sacrifices accuracy and fixed sparsity overlooks timestep differences.Existing methods commonly apply similar sparsity across denoising steps despite distinct timestep characteristics.Source: §2.2 · pp. 3–4

3 Motivation

The motivation identifies three shortcomings in sparse attention: clustering objectives can misalign with query-key interaction error, fixed sparsity overlooks denoising dynamics, and irregular clusters waste kernel work.

3.1 Independent Clustering Mismatches Q-K Interaction Objectives

  • Independent feature-space clustering does not align with minimizing query-key interaction error, while explicitly clustering interactions is too costly for long sequences.iCATS instead minimizes the interaction error with a low-complexity scheme that avoids constructing the full matrix.Source: §3.1 · p. 4

3.2 Fixed Sparsity Overlooks Denoising Dynamics

  • Fixed top-p sparsity assumes approximation errors matter uniformly, but denoising recursively propagates errors whose effects on the final output vary by timestep.SNR-guided sparsity is motivated as a way to allocate a fixed computation budget according to denoising progress.Source: §3.2 · p. 4

3.3 Varying Cluster Sizes Incur Tail Overhead

  • Irregular cluster sizes leave execution blocks partially filled, and query-side padding repeats across selected key/value blocks, increasing overhead.This motivates tail merging to reduce tail execution overhead and improve hardware efficiency.Source: §3.3 · p. 5

4 Method

iCATS combines interaction-aware clustering, timestep-adaptive sparsity, and tail merging to optimize importance estimation, sparse-mask construction, and execution. Its pipeline uses the resulting cluster-level scores to select attention regions before restructuring query clusters for sparse-kernel execution.

4.1 Interaction-Aware Clustering

  • Interaction-aware clustering uses query-key dot-product interactions, reformulated as a quadratic form in the original feature space to avoid constructing the full interaction matrix.At the first sparse timestep, query clusters are initialized in feature space; later query clustering uses the previous key-cluster metric, and the updated query clusters guide key clustering.Source: §4.1 · pp. 5–6

4.2 SNR-Guided Sparsity

  • SNR-guided sparsity adapts top-p across denoising timesteps, using the logSNR stage pattern to reduce top-p more smoothly in the middle while keeping the early and late budgets relatively stable.Dense attention is retained before the first sparse timestep; the SNR-derived schedule can be computed offline from the predefined noise schedule.Source: §4.2 · pp. 6–7

4.3 Tail-Merging Strategy

  • Tail merging restructures query clusters to reduce padding and scheduling overhead while preserving the sparse attention pattern as much as possible.Clusters in the same attention head that attend to the same key clusters can be merged without changing that pattern; remaining tail tokens are reassigned using key-mask similarity under size and alignment constraints.Source: §4.3 · p. 7

5 Experiments and Analysis

Experiments compare iCATS with SVG and SVG2 across video-generation models and tasks, finding improved quality-efficiency trade-offs, cross-hardware performance comparable to H100, and a preference for iCATS-Base in human evaluation.

5.1 Setup

  • The evaluation covers three video-generation DiTs and 5-second 720p generation, using PSNR, SSIM, LPIPS, VBench, speedup, and sparsity metrics.Comparisons use official SVG and SVG2 implementations; experiments are generally run on eight NVIDIA H100 GPUs.Source: §5.1 · p. 8

5.2 Quality & Efficiency Evaluation

  • iCATS-Base and iCATS-Turbo achieve better quality-efficiency trade-offs than SVG and SVG2 on HunyuanVideo-T2V-13B and Wan2.1-T2V-14B.The evaluation also reports the same trend on Wan2.1-I2V-14B.Source: §5.2 · p. 8, Table 1 · p. 8
  • Interaction-aware clustering achieves 1.7 points higher selection overlap and 36% lower attention-output MSE than independent clustering, with gains across sampled timesteps, layers, and heads.The diagnostic compares both clustering schemes with exact dense attention on Wan2.1-T2V-14B.Source: §5.2 · p. 9, Table 2 · p. 9
  • Tail merging reduces execution-block counts most at early timesteps, with the reduction gradually decreasing during denoising.On an H100, merging takes about 10 seconds per generation against 1,222 seconds total inference time.Source: §5.2 · p. 9, Figure 5 · p. 9

5.3 Cross-Hardware Validation

  • iCATS on Wan2.1-T2V-14B achieves performance comparable to its H100 GPU performance on the T-Head Zhenwu M890P3.Sparse kernel operators were adapted to the accelerator platform.Source: §5.3 · p. 9, Table 3 · p. 9

5.4 Human Evaluation

  • On evaluated Wan2.1-T2V-14B samples, iCATS-Base receives 54.1% of votes versus 18.6% for SVG2, while 27.3% indicate similar quality, showing a human preference.Source: §5.4 · p. 10, Table 4 · p. 10

5.5 Ablation Study

  • Interaction-aware clustering and SNR-guided sparsity improve quality over the Anchor configuration, and their combination yields further gains; adding tail merging gives the best overall trade-off.Tail merging raises speedup from 1.56× to 1.61× while improving quality metrics, despite slightly reducing sparsity.Source: §5.5 · p. 10, Table 5 · p. 10

6 Conclusion & Limitation

The conclusion presents iCATS as a training-free framework that optimizes importance estimation, mask construction, and hardware execution, while identifying clustering and masking-kernel optimization as future work. Its SNR-guided schedule varies computation density along denoising.

6 Conclusion & Limitation

  • iCATS combines interaction-based importance estimation, SNR-guided sparsity scheduling, and tail merging for training-free DiT video-generation acceleration.The paper characterizes its performance across various DiT models as state of the art.Source: §6 · p. 10
  • The clustering procedure is not optimized for multi-GPU execution, and the token-level sparse masking kernel remains to be further optimized.The authors leave multi-GPU clustering optimization for future work.Source: §6 · p. 10

A SNR-Guided Sparsity

  • iCATS keeps full computation during warmup, reduces density through the middle denoising stage, and stabilizes it late, unlike SVG2's fixed-density baseline.The comparison is shown for Wan2.1-T2V-14B; iCATS-Turbo uses a more aggressive schedule.Source: A SNR-Guided Sparsity · p. 14, Figure 1 · p. 14

B Detailed Experimental Settings

The experimental settings compare iCATS with SVG and SVG2 using their official configurations, and evaluate Base and Turbo variants with model-specific clustering and sparsity parameters.

B Detailed Experimental Settings

  • SVG and SVG2 use their official open-source implementations and default settings across the evaluated models.Defaults include warmup timesteps, warmup layers, and clustering iterations.Source: B Detailed Experimental Settings · p. 14
  • The Base and Turbo variants are configured to match prior methods in speed or sparsity, with warmup and clustering settings following each model's baseline configuration.The appendix summarizes those configurations in Table 1.Source: B Detailed Experimental Settings · p. 14, Table 1 · p. 14
  • Query/key cluster counts and pmin and τ settings vary by model and variant.HunyuanVideo uses 500/1000 clusters; Base pmin = 0.7, τ = 0.98, and Turbo pmin = 0.65, τ = 0.97. Wan2.1-T2V uses 400/1000; Base pmin = 0.7, τ = 0.999, and Turbo pmin = 0.65, τ = 0.97. Wan2.1-I2V uses 400/1000; Base pmin = 0.8, τ = 0.999, and Turbo pmin = 0.75, τ = 0.98.Source: B Detailed Experimental Settings · p. 14

C Additional VBench Results

The additional VBench results show that iCATS preserves quality across HunyuanVideo text-to-video and Wan2.1 text-to-video and image-to-video. Its scores match or closely approach dense attention while remaining competitive with sparse baselines.

C Additional VBench Results

  • Table 2 reports subject consistency, background consistency, motion smoothness, aesthetic quality, imaging quality, and average VBench score across benchmark models.These detailed results complement the average scores reported in the main paper.Source: C Additional VBench Results · p. 15, Table 2 · p. 15
  • Both HunyuanVideo-T2V-13B variants score 0.844 on average, matching dense attention and matching it on subject consistency, background consistency, motion smoothness, and imaging quality.Against sparse baselines, both variants achieve the strongest subject consistency and tie for best background consistency and motion smoothness.Source: C Additional VBench Results · p. 15, Table 2 · p. 15
  • On Wan2.1-T2V-14B, iCATS-Base scores 0.838 on average versus SVG's 0.836 and SVG2's 0.837; iCATS-Turbo scores 0.837, matching SVG2.iCATS achieves the strongest imaging quality among the sparse variants.Source: C Additional VBench Results · p. 15, Table 2 · p. 15
  • On Wan2.1-I2V-14B, both iCATS variants score 0.846 on average, exceeding SVG (0.844) and SVG2 (0.845) and staying close to dense attention (0.847).Both variants preserve dense-attention motion smoothness; Base matches its imaging quality, while Turbo matches its background consistency.Source: C Additional VBench Results · p. 15, Table 2 · p. 15

D Attention Fidelity Breakdown

Interaction-aware clustering outperforms independent Euclidean clustering on both attention-fidelity metrics across sampled timesteps, layers, and heads. The tail-merging algorithm combines compatible clusters to reduce block padding while largely preserving sparsity structure.

D Attention Fidelity Breakdown

  • Interaction-aware clustering consistently outperforms independent Euclidean clustering on both attention-fidelity metrics across sampled timesteps, layers, and heads.Table 3 reports entries as interaction-aware (ours) / Euclidean, with MSE values in units of 10^-4.Source: D Attention Fidelity Breakdown · p. 15, Table 3 · p. 16

E.1 Algorithm Overview

  • The algorithm first merges clusters with identical key-cluster masks within each head, summing their sizes and remapping token labels.It packs masks into compact integers to identify identical masks.Source: §E.1 · p. 17
  • It then moves tail tokens only between mutually closest clusters whose tails fit one block, unions their masks, and removes empty clusters after two rounds.This improves block utilization while largely preserving the original sparsity structure.Source: §E.1 · p. 17

E.2 Implementation Details

Tail merging reduces the number of executed FlashInfer blocks, with the average reduction varying over denoising. The reported reduction is larger early and declines as sparsity increases and key/value sequence lengths shrink.

E.2 Implementation Details

  • Executed FlashInfer blocks fall by about 25% on average early in denoising, declining to around 7% as denoising progresses.The min-max range shows that block reduction varies across sparse-attention layers at each timestep.Source: §E.2 · p. 17, Figure 2 · p. 17
  • FlashInfer uses query-side blocks of 128 and key/value-side blocks of 96 in the reported implementation.The reduction is measured in executed blocks under iCATS-Base with SNR-guided sparsity configured by τ = 0.999 and pmin = 0.7.Source: §E.2 · p. 17
  • As the SNR-guided schedule increases sparsity, key/value sequence lengths decrease and the average block-count reduction gradually falls.The reported reduction varies across sparse-attention layers at each denoising timestep.Source: §E.2 · p. 17

F Hyperparameter Sensitivity and Transferability

A configuration transferred from HunyuanVideo-Base improves quality over SVG2 on both Wan2.1 tasks at comparable speedup. Sensitivity tests on Wan2.1-T2V-14B show that increasing sparsity and speedup gradually reduces reconstruction quality across the tested schedule settings.

F Hyperparameter Sensitivity and Transferability

  • The transfer reuses HunyuanVideo-Base query/key cluster counts and SNR-guided sparsity hyperparameters on both Wan2.1 tasks without retuning.Warmup timesteps, warmup layers, and clustering iterations follow each target model's baseline settings; evaluation uses 100 randomly sampled prompts.Source: F Hyperparameter Sensitivity and… · p. 17
  • The transferred configuration beats SVG2 on PSNR, SSIM, and LPIPS for both Wan2.1 tasks at comparable speedup.For Wan2.1-T2V-14B, it reaches 28.656 dB PSNR versus SVG2's 25.411 dB, only 0.404 dB below Wan-tuned iCATS-Base; for Wan2.1-I2V-14B, PSNR rises from 29.068 to 29.810 dB with slightly higher speedup.Source: F Hyperparameter Sensitivity and… · p. 17, Table 4 · p. 18
  • Sensitivity tests change τ or pmin independently from Wan2.1-T2V-14B Base settings while keeping Wan-specific cluster counts fixed.The tested changes decrease τ to 0.99, 0.98, or 0.97, or pmin to 0.6.Source: F Hyperparameter Sensitivity and… · p. 17
  • Across tested τ values, sparsity rises from 60.5% to 66.4% and speedup from 1.54× to 1.58×, with gradual reconstruction-quality reduction.All tested settings retain better PSNR, SSIM, and LPIPS than SVG2.Source: F Hyperparameter Sensitivity and… · p. 18, Table 5 · p. 18

G Additional Visualizations

Additional visual comparisons on Wan2.1-T2V-14B show that iCATS preserves high visual fidelity and produces results visually close to dense attention across diverse scenes and motion patterns.

G Additional Visualizations

Related papers

Loading 2610.11302v1…