PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Video Generation with Sliding Tile Attention

Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhengzhong Liu, Hao Zhang

arXiv:2502.04507v3cs.CV

TL;DR

Full 3D attention makes video diffusion computationally expensive. STA uses tile-based local attention, accelerating attention by >10× and end-to-end video generation by up to 3.53× with no or minimal quality loss.

  • Problem

    Full 3D attention makes video DiTs computationally expensive; HunyuanVideo requires 16 minutes to generate a 5-second 720p video even with FA3 on an H100 GPU.

    Source: §1 · p. 1

  • Method

    STA uses tile-based sliding-window attention, grouping queries so each tile attends to shared local key tiles and preserving 3D locality.

    Source: §3.1 · p. 4

  • Results

    STA accelerates attention by >10× and end-to-end video generation by up to 3.53× with no or minimum quality loss.

    Source: §1 · p. 2

  • Takeaways & Limitations

    STA is conceptually orthogonal to caching and consistency distillation, though exploring their combined effectiveness is future work.

    Source: §6 · p. 9

    Other memory-bound operations, such as LayerNorm and modulation, likely contribute to inference overhead, preventing full FLOPs reductions from translating directly into speedup.

    Source: §5 · p. 9

Abstract

from arXiv · show

Diffusion Transformers (DiTs) with 3D full attention power state-of-the-art video generation, but suffer from prohibitive compute cost -- when generating just a 5-second 720P video, attention alone takes 800 out of 945 seconds of total inference time. This paper introduces sliding tile attention (STA) to address this challenge. STA leverages the observation that attention scores in pretrained video diffusion models predominantly concentrate within localized 3D windows. By sliding and attending over the local spatial-temporal region, STA eliminates redundancy from full attention. Unlike traditional token-wise sliding window attention (SWA), STA operates tile-by-tile with a novel hardware-aware sliding window design, preserving expressiveness while being hardware-efficient. With careful kernel-level optimizations, STA offers the first efficient 2D/3D sliding-window-like attention implementation, achieving 58.79% MFU. Precisely, STA accelerates attention by 2.8-17x over FlashAttention-2 (FA2) and 1.6-10x over FlashAttention-3 (FA3). On the leading video DiT, HunyuanVideo, STA reduces end-to-end latency from 945s (FA3) to 685s without quality degradation, requiring no training. Enabling finetuning further lowers latency to 268s with only a 0.09% drop on VBench. We make our codebase public at https://github.com/hao-ai-lab/FastVideo.

1. Introduction

Full 3D attention makes video DiTs computationally expensive, even though pretrained models concentrate attention within localized spatial-temporal regions. STA exploits this locality with tile-based, hardware-aware attention to reduce inference cost while retaining generation quality.

1. Introduction

  • A local window covering 15.52% of HunyuanVideo’s token space captures 70% of total attention scores, revealing substantial 3D locality despite full-attention training.Queries assign higher attention scores to spatially and temporally nearby keys.Source: §1 · pp. 1–2
  • Existing higher-order sliding-window methods NATTEN (Hassani et al., 2023) and CLEAR (Liu et al., 2024) fail to convert FLOP reductions into proportional wall-clock speedups because irregular masks waste computation and GPU utilization.Their masking overhead makes the computation pattern GPU-unfriendly.Source: §1 · p. 2
  • STA slides over contiguous spatial-temporal tiles and uses producer warpgroups for sparse loading and consumer warpgroups for dense computation, avoiding explicit attention masking.The design enables memory access and parallelism while achieving wall-clock speedups proportional to sparsity.Source: §1 · p. 2
  • 501s generates a 5-second 720P HunyuanVideo clip with STA, versus 1496s with FlashAttention-2 and 945s with FlashAttention-3; finetuning yields 2.43–3.53× end-to-end speedup over FlashAttention-3.The plug-and-play STA result has no or minimal quality loss; finetuning uses more radical attention sparsity.Source: §1 · p. 2
  • Head specialization gives different attention heads distinct locality scales, and STA profiles each head to select window sizes automatically while balancing efficiency and quality.The specialization is prompt-agnostic.Source: §1 · p. 2, Figure 3 · p. 2

2. Problem

Video DiTs use costly 3D full attention, while straightforward sliding-window or factorized alternatives either waste GPU work or fail to preserve 3D locality.

2.1. Attention in Video DiTs

  • Because a video latent of shape (L, L, L) becomes a sequence of length L3, full 3D attention grows prohibitively expensive as video resolution or duration increases.The section assumes cubic window, tile, and video sizes for notation, while the approach extends to non-cubic configurations.Source: §2.1 · p. 3, §2 · p. 3
  • Although video diffusion models exhibit strong 3D locality, directly applying token-wise SWA does not fully utilize GPU computation in high-dimensional settings [Peebles & Xie, 2023].Stacking SWA layers expands the receptive field beyond the fixed window, but the block-level inefficiencies motivate minimizing mixed blocks while maintaining locality.Source: §2.1 · p. 3, §2.2 · p. 3

2.2. Inefficiency of Sliding Window Attention

  • SWA preserves locality by limiting each query to a fixed window, but in high-dimensional data its zigzag attention pattern creates many mixed blocks that waste computation and incur mask-evaluation overhead.Mixed blocks require full block computation before masked scores are discarded; calculating even a simple causal mask has been reported to add 15% overhead in FlexAttention.Source: §2.2 · p. 3

2.3. Alternative Methods for 3D Attention

  • Factorized spatial-temporal attention has been largely superseded by full 3D attention because it can miss interactions between tokens offset in both dimensions [Wang et al., 2023; Ma et al., 2024].For example, a query at (1, 1, 1) can attend to (1, X, X) or (X, 1, 1), but not (2, 2, 2), despite their spatial closeness.Source: §2.3 · pp. 3–4

3. Methods

STA restructures local video attention into tile-aligned dense blocks, and the paper describes both GPU-oriented kernel implementation and training-free or finetuned ways to apply it to video DiTs.

3.1. SLIDING TILE ATTENTION

  • STA groups queries and keys into tiles so queries in each tile share a local window, forming dense FlashAttention blocks and eliminating mixed blocks.For 3D attention, tiles are non-overlapping (T, T, T) regions with B = T^3; the sequence is flattened so each tile’s tokens have consecutive indices, and the window slides by one tile.Source: §3.1 · p. 4

All remaining blocks are empty and there are no mixed

  • STA kernels skip empty blocks and avoid unnecessary intra-block masks, while asynchronous data warpgroups load only needed KV blocks as compute warpgroups process query blocks.Inter-block mask logic is disaggregated from compute, allowing mask calculation and data selection to overlap with computation.Source: All remaining blocks are empty and… · p. 5

3.2. Applying STA to Video Diffusion Model

  • Training-free STA searches head-specific windows using mask-search loss averaged over 16 prompts, retains full attention for the initial T0 timesteps, then applies STA.The search exploits locality patterns that are largely consistent across prompts.Source: §3.2 · p. 5
  • Finetuning adapts a high-sparsity fixed window, using attention distillation, final-layer alignment, and flow-matching data losses.The adaptation took 8 hours on 8 H100s; although each layer is local, stacked layers expand the receptive field to support globally coherent videos.Source: §3.2 · pp. 5–7

4. Experiments

Experiments evaluate STA’s efficiency and video quality on HunyuanVideo, using human evaluation and automated metrics, and apply STA to image super-resolution with FLUX.

4. Experiments

  • The evaluation uses HunyuanVideo outputs of 117 frames at 1280×768 resolution and compares STA with CLEAR, NATTEN, Swin, and adapted ∆-DiT.Video quality assessment emphasizes human evaluation and also reports VBench, SSIM, PSNR, and CD-FVD; efficiency is measured with MFU and wall-clock latency.Source: §4 · p. 7

4.2. Human Evaluations

  • STA-t-2.43x wins 70.0% versus ∆-DiT-1.8x’s 11.0%, while STA-tf-1.89x wins 66.5% versus ∆-DiT-1.36x’s 10.0%.Against original HunyuanVideo, STA-tf-1.89x ties in 83.0% of comparisons while providing 1.89× speedup.Source: Figure 7 · p. 7, §4.2 · p. 8

4.1. Efficiency of SLIDING TILE ATTENTION

  • STA raises MFU from 8.20% to 41.03% in FlexAttention and reaches 10.45× speedup over full attention with ThunderKittens optimization.STA slides tile-by-tile rather than token-by-token; CLEAR slows down 0.86×, vanilla NATTEN slows down 0.85×, and tiled NATTEN achieves 1.27× speedup.Source: §4.1 · p. 7

4.3. Training-free Results

  • At 50 steps, training-free STA scores 87.67 SSIM and 28.76 PSNR versus ∆-DiT’s 72.86 SSIM and 18.09 PSNR.STA’s CD-FVD is 66.12 versus ∆-DiT’s 122.74, with lower scores better; STA also maintains higher fidelity at 25 and 10 steps.Source: Table 3 · p. 8, §4.3 · p. 8

4.4. Finetuning Results

  • STA’s wt=(3,3,3) configuration yields 91.00% attention sparsity, 5.76× FLOPs reduction, and 3.53× latency reduction; its VBench score rises from 80.58% training-free to 82.62% after fine-tuning.Full attention scores 82.71% on VBench. CLEAR and Tiled NATTEN retain reasonable quality but increase end-to-end latency, while Swin’s rigid windows degrade quality.Source: Table 4 · p. 9, §4.4 · p. 9

4.5. Results on Image Super-Resolution

  • STA on FLUX achieves comparable generation quality to CLEAR while offering significantly higher efficiency for image super-resolution with SDEdit.The experiments are reported in Appendix E.Source: §4.5 · p. 9

5. Related Work

Related work covers attention and diffusion-step acceleration; STA is largely complementary to methods that reduce sampling steps, while memory-bound operations can limit realized speedups.

5. Related Work

6. Conclusion and Future Work

Sliding Tile Attention accelerates video diffusion through an optimized high-order sliding-window-like attention kernel that preserves locality and GPU efficiency, with minimal or no quality loss. The method is orthogonal to caching and consistency distillation, whose combination the authors plan to explore.

6. Conclusion and Future Work

  • Sliding Tile Attention accelerates video generation with minimal or no quality loss.Experiments demonstrate the acceleration without requiring a stronger claim about the magnitude of quality change.Source: §6 · p. 9
  • An optimized kernel enables efficient GPU execution of high-order sliding-window-like attention while preserving locality.The method applies sliding tile attention to accelerate video diffusion models.Source: §6 · p. 9
  • The authors plan to explore combining Sliding Tile Attention with orthogonal techniques such as caching and consistency distillation for further efficiency gains.Source: §6 · p. 9

Impact Statement · A. Further Details of SLIDING TILE ATTENTION · B. Finetuning Details

The sections describe STA’s tile-based masking and locality-preserving token ordering, state the work’s expected accessibility benefits and potential misuse risks, and detail the HunyuanVideo finetuning setup.

Impact Statement

  • The work presents faster video generation as a way to make video AI more practical for researchers and developers with limited computing resources.It notes possible misuse and says content detection and watermarking can help mitigate the risks.Source: Impact Statement · p. 10

A. Further Details of SLIDING TILE ATTENTION

  • STA maps query and key coordinates into tiles, centers windows in tile coordinates, and selects neighboring keys by tile distance.This tile-based mask differs from 3D NATTEN’s token-coordinate window mask, whose center shifts inward near video edges.Source: A. Further Details of SLIDING TILE… · p. 13
  • STA’s tile ordering keeps neighboring tokens’ sequence IDs consecutive, so attention between tiles forms dense blocks.The passage describes a 2D illustration and says the strategy extends naturally to 3D.Source: A. Further Details of SLIDING TILE… · p. 13

B. Finetuning Details

  • Finetuning uses 2,000 synthetic HunyuanVideo videos at 1280×768 resolution with 117 frames, with prompts sourced from Mixkit.VAE latents and text-encoder states are precomputed to reduce memory use and accelerate training.Source: B. Finetuning Details · p. 13
  • Training finetunes for 1,600 steps with batch size 2 and learning rate 2e-5, using loss coefficients α = 1, β = 0.5, and γ = 0.5.The loss is from Eq. (5). Guidance scales 1 and 6 alternate on odd and even steps to avoid overfitting to one scale.Source: B. Finetuning Details · p. 13
  • Training runs for 8 hours on 8 H100 GPUs with FSDP and context parallelism; inference uses sequence parallelism.Source: B. Finetuning Details · p. 13

C. Further Details of Baselines

The section describes Swin and CLEAR local-attention baselines and ∆-DiT caching, including how their mechanisms and evaluation settings are configured.

C. Further Details of Baselines

  • Swin Transformer alternates shifted local windows for cross-window exchange, but can disrupt connectivity within a layer; here, its windows shift every other HunyuanVideo layer.Its alternating partitioning strategy connects regions across layers, though tokens separated into different windows may not attend to each other in one layer.Source: C. Further Details of Baselines · pp. 13–14
  • CLEAR (Liu et al., 2024) replaces full attention with circular radius-r windows, restricting each query to local key-value tokens while retaining scaled dot-product attention.The authors implement CLEAR with FlexAttention.Source: C. Further Details of Baselines · p. 14
  • For ∆-DiT, the 1.36× and 1.8× speedup settings use Nc = 24, N = 3, b = 24 and Nc = 28, N = 6, b = 24, respectively.The method caches feature offsets rather than full feature maps, using later-block residuals for early-step sampling and earlier-block residuals for later steps; hyperparameters are selected for a given speedup budget.Source: C. Further Details of Baselines · p. 14

D. Results on Wan 2.1

On Wan 2.1, STA maintains comparable performance across key metrics at the same sparsity level, while its end-to-end speedup is reduced for the shorter evaluated videos.

D. Results on Wan 2.1

  • On Wan 2.1, STA achieves comparable performance across key metrics at the same sparsity level, but the 69-frame videos yield a reduced end-to-end speedup.The videos use the same resolution as the prior evaluation but are shorter.Source: D. Results on Wan 2.1 · p. 14

E. Results on Image Super-Resolution · F. More Experiment Results · F.1. Kernel Performance

The section reports FLUX image-superresolution evaluation on COCO-2014 captions and additional sparse-kernel benchmarks. At around 56% sparsity, lower sparsity generally corresponds to higher MFU, while the findings in Table 2 remain unchanged.

E. Results on Image Super-Resolution

  • Image super-resolution results evaluate FLUX (Black-Forest, 2023) on 1,000 captions randomly sampled from the COCO-2014 (Lin et al., 2015) validation dataset.The supplied caption identifies Table 6's scope but does not provide its results.Source: Table 6 · p. 14

F.1. Kernel Performance

  • At around 56% sparsity, sparse attention kernels were benchmarked; at lower sparsity, they generally achieve higher MFU, while Table 2 findings remain unchanged.The benchmark results are presented in Table 7, but its values are not included in the supplied passage.Source: §F.1 · p. 14

F.2. Detailed VBench Results · G. Qualitative Examples

STA improves video-quality dimensions over some sparse-attention alternatives, while sparsity creates a tradeoff between quality-related and semantic-alignment metrics. Qualitative examples show that fine-tuned STA retains high generation quality despite minor output shifts, whereas ∆-DiT outputs are generally less sharp.

F.2. Detailed VBench Results

  • With higher sparsity, training-free STA degrades on quality-related metrics such as temporal flickering and motion smoothness, while Appearance Style, Color, and Spatial Relationships improve.The paper attributes the semantic-alignment gains to text embeddings playing a larger role as spatial-temporal attention is sparsified.Source: §F.2 · p. 15
  • Trained STA improves video-quality metrics over untrained STA while maintaining semantic coherence at comparable levels.The passage presents this as improved low-level visual fidelity without compromising text-video alignment.Source: §F.2 · p. 15
  • STA surpasses Swin attention on Imaging Quality and Multiple Objects, and scores comparably or better than CLEAR and Tiled NATTEN on the reported dimensions.The detailed comparisons are presented in Tables 8 and 9; the supplied passages do not provide specific score values.Source: §F.2 · pp. 14–15, Table 8 · p. 16, Table 9 · p. 16

G. Qualitative Examples

Related papers

Loading 2502.04507v3…