Source-linked AI summary
Fast Video Generation with Sliding Tile Attention
Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhengzhong Liu, Hao Zhang
TL;DR
Video diffusion Transformers are bottlenecked by the cost of full 3D attention. The paper introduces hardware-aware Sliding Tile Attention, which exploits localized attention through tile-based computation. STA achieves substantial kernel and end-to-end speedups with minimal or no quality loss, while remaining subject to scope limitations identified for alternative 3D attention methods and residual inference overhead.
Problem
Full 3D attention in video DiTs has quadratic computational cost, making high-resolution and long-duration video generation prohibitively slow.
Method
STA replaces token-wise sliding with hardware-aware tile-by-tile attention, using dense consumer computation and producer-managed sparse masking to exploit 3D locality.
Results
STA achieves 58.79% MFU, accelerates attention by more than 10×, and speeds end-to-end video generation by up to 3.53× with minimal or no quality loss.
Takeaways & Limitations
STA provides an efficient sliding-window-like implementation for video diffusion and is orthogonal to caching and consistency distillation for potential combination.
Takeaways & Limitations
Alternating spatial and temporal attention can miss tokens offset in both dimensions, while other inference operations may prevent FLOP reductions from translating directly into speedups.
Abstract
from arXiv · showhide
Diffusion Transformers (DiTs) with 3D full attention power state-of-the-art video generation, but suffer from prohibitive compute cost -- when generating just a 5-second 720P video, attention alone takes 800 out of 945 seconds of total inference time. This paper introduces sliding tile attention (STA) to address this challenge. STA leverages the observation that attention scores in pretrained video diffusion models predominantly concentrate within localized 3D windows. By sliding and attending over the local spatial-temporal region, STA eliminates redundancy from full attention. Unlike traditional token-wise sliding window attention (SWA), STA operates tile-by-tile with a novel hardware-aware sliding window design, preserving expressiveness while being hardware-efficient. With careful kernel-level optimizations, STA offers the first efficient 2D/3D sliding-window-like attention implementation, achieving 58.79% MFU. Precisely, STA accelerates attention by 2.8-17x over FlashAttention-2 (FA2) and 1.6-10x over FlashAttention-3 (FA3). On the leading video DiT, HunyuanVideo, STA reduces end-to-end latency from 945s (FA3) to 685s without quality degradation, requiring no training. Enabling finetuning further lowers latency to 268s with only a 0.09% drop on VBench. We make our codebase public at https://github.com/hao-ai-lab/FastVideo.
1. Introduction
Video DiTs achieve strong generation quality with 3D full attention, but its quadratic cost makes attention the dominant bottleneck. STA exploits localized attention and hardware-aware tile processing to accelerate HunyuanVideo while preserving quality.
- Motivation: 16 minutes are required to generate a 5-seconds 720p HunyuanVideo clip with FlashAttention 3 on an H100.Attention computation dominates the overall inference cost.
- Attention locality: 70% of HunyuanVideo’s total attention score lies within a local window covering 15.52% of the token space.This locality appears despite training with full 3D attention.
- Limitations of prior methods: Existing 2D/3D sliding-window methods fail to convert FLOP reductions into proportional wall-clock speedups because irregular masks waste computation and reduce GPU utilization.Their higher-order sliding patterns also incur substantial masking overhead.
- Sliding Tile Attention: STA slides over spatial-temporal tiles rather than tokens, using hardware-aware execution to preserve locality while eliminating explicit attention masking during computation.Producer warpgroups manage sparse masking while consumer warpgroups perform dense attention computation.
- Results: 58.79% MFU is achieved by STA’s optimized kernel, while attention speedups exceed 10× and end-to-end video-generation speedups reach 3.53× with minimal or no quality loss.The method also profiles head-specific window sizes to balance efficiency and quality.
2. Problem
High-dimensional video attention becomes expensive because 3D sequences grow cubically, while conventional 2D/3D sliding-window implementations waste computation on irregular mixed blocks. Efficient sparse attention therefore needs to preserve locality while reducing mixed-block and masking overhead.
- 2. Problem: 3D video attention flattens an (L, L, L) latent into L3 tokens, so small increases in resolution or duration sharply increase computational burden.Applying 3D attention to high-resolution, long-duration videos becomes prohibitively expensive.
- 2. Problem: SWA restricts each query to keys within a fixed local window, preserving locality while reducing attention computation.Stacking multiple SWA layers naturally expands the receptive field beyond one window.
- 2. Problem: FlashAttention classifies masked blocks as dense, empty, or mixed, but mixed blocks still require full block computation before masked values are discarded.Empty blocks can be skipped, whereas mixed blocks introduce significant overhead.
- 2. Problem: SWA beyond one-dimensional sequences has two major inefficiencies: mixed blocks do not reduce FLOPs, and mask evaluation adds overhead.These inefficiencies motivate minimizing mixed blocks while maintaining SWA locality.
- 2. Problem: Alternating spatial and temporal attention can miss interactions offset in both dimensions, such as between tokens at (1, 1, 1) and (2, 2, 2).This can disrupt the 3D locality pattern identified for video diffusion models.
3. Methods
STA restructures sliding-window attention around tiles so queries share key groups, producing GPU-friendly dense and empty blocks instead of irregular mixed blocks. It supports training-free replacement or lightweight finetuning, with head-specific window selection and receptive fields that expand across transformer layers.
- Sliding Tile Attention: STA groups consecutive tokens into spatial-temporal tiles so each query tile attends to shared key tiles within its local window.The tile area is matched to the FlashAttention block size, preserving local 3D attention while creating structured computation.
- Kernel-level optimization: STA kernels use FlexAttention, ThunderKittens, and FlashAttention3 optimizations to skip empty blocks and reduce unnecessary intra-block mask processing.The implementation separates inter-block mask logic from compute kernels and uses a producer-consumer execution pattern.
- Applying STA to Video Diffusion Models: STA can directly replace pretrained video-DiT attention without training, while lightweight finetuning enables greater sparsity.The paper reports a finetuning setup requiring 8 hours on 8 H100 GPUs, with attention distillation, final-layer, and data losses.
- Applying STA to Video Diffusion Models: A profiling procedure selects window sizes per attention head, exploiting head-specific locality patterns that remain largely consistent across prompts.The method searches mask patterns on a small number of prompts and applies the selected configuration to other prompts.
- Applying STA to Video Diffusion Models: Stacked transformer layers expand the receptive field beyond each local attention window, enabling globally coherent video generation.The local restriction is therefore applied per layer rather than as a fixed global receptive-field limit.
4. Experiments
Experiments evaluate STA’s efficiency and quality on HunyuanVideo, including training-free and fine-tuned settings, against sparse-attention and caching baselines. STA combines substantial speedups with competitive or preserved video quality.
- Experimental setup: STA is evaluated on HunyuanVideo at 1280×768 resolution with 117 frames, alongside FLUX image superresolution experiments.Baselines include CLEAR, NATTEN, Swin, and ∆-DiT.
- Kernel efficiency: CLEAR reduces TFLOPs to 15.65 but slows down by 0.86×, while vanilla NATTEN slows by 0.85× and optimized tiled NATTEN reaches only 1.27× speedup.Swin exceeds 40% MFU and 60% kernel efficiency but is not sliding-window attention.
- Kernel efficiency: STA improves MFU from 8.20% to 41.03% over Tiled NATTEN in FlexAttention and reaches 10.45× speedup over full attention with its optimized kernel.At 58.33% sparsity, STA achieves 2.37× speedup.
- Human evaluation: STA-t-2.43x achieves a 70.0% win rate versus 11.0% for ∆-DiT-1.8x, while STA-tf-1.89x reaches 66.5% versus 10.0% for ∆-DiT-1.36x.The human evaluation uses 200 MovieGen Bench prompts.
- Training-free results: At 50 sampling steps, training-free STA improves SSIM to 87.67 from 72.86 and PSNR to 28.76 from 18.09 versus ∆-DiT.STA’s CD-FVD is 66.12 versus 122.74 for ∆-DiT, where lower is better.
- Fine-tuning results: STA with wt=(3,3,3) achieves 91.00% attention sparsity, a 3.53× latency reduction, and VBench scores of 80.58% training-free and 82.62% after fine-tuning.The corresponding FLOPs reduction is 5.76×.
- 2D evaluation: FLUX with STA provides image superresolution quality comparable to CLEAR while offering significantly higher efficiency.
5. Related Work
Related work covers linear-attention approaches for reducing quadratic softmax attention and methods that accelerate diffusion sampling. STA is presented as complementary to these approaches.
- Attention acceleration: Linear attention methods use kernel or gate functions to decompose quadratic softmax attention and achieve linear complexity, but have not succeeded in video DiTs.
- Attention acceleration: Other recent video-DiT acceleration directions include quantized attention and sparse attention with different designs.
- Diffusion sampling: Sampling acceleration methods reduce diffusion steps through faster numerical solvers, consistency distillation, adversarial distillation, and other distillation approaches.
- Complementarity: STA is described as largely complementary to caching and consistency-distillation methods.
6. Conclusion and Future Work
The paper concludes that STA provides efficient high-order sliding-window-like attention for video diffusion while preserving locality and video quality. It identifies combination with other acceleration methods as future work.
- Conclusion: STA’s optimized attention kernel enables efficient GPU execution while preserving the locality property of video diffusion attention.
- Conclusion: Experiments show that STA accelerates video generation with minimal or no quality loss.
- Future work: The authors plan to explore combining STA with caching and consistency distillation for further efficiency gains.
Impact Statement
The impact statement frames STA as a way to reduce video-generation time and make generation more practical under limited computing resources. It also notes potential misuse concerns and describes the method’s implementation and training context.
- Potential benefits: STA reduces video-generation time while maintaining output quality, making generation more practical for researchers and developers with limited computing resources.
- Potential benefits: The stated applications include AI-driven video use in creative industries and education.
- Risks: The paper acknowledges that faster video generation could potentially enable misuse.
- Method context: STA maps queries and keys into tile coordinates, selects neighboring keys by tile distance, and preserves locality through tile-based token ordering.
- Training context: Fine-tuning uses 2,000 synthetic HunyuanVideo videos at 1280×768 resolution with 117 frames for 1,600 steps.
C. Further Details of Baselines
The baselines include shifted-window, circular-window, and residual-caching approaches, alongside visualizations of conventional and STA' sequence flattening and 2D sliding-window attention.
- Swin Transformer: Swin attention partitions images into local windows and shifts them every other layer to enable cross-window information exchange.Applied here to HunyuanVideo, this train-from-scratch approach can disrupt local connectivity within a single layer.
- STA' and SWA visualizations: STA' uses a tile-based sequence flattening strategy, contrasted with conventional zigzag flattening for a (9, 9) image with (3, 3) tiles.A separate visualization presents the 2D sliding-window attention pattern.
- CLEAR: CLEAR replaces full attention with circular windows in which each query attends only to key-value tokens within radius r.It maintains the scaled dot-product attention formula and is implemented with FlexAttention.
- ∆-DiT: ∆-DiT accelerates video generation by caching feature offsets with staged residual reuse across early and later sampling steps.Its key parameters are cache interval N, cached blocks Nc, and timestep boundary b; a 1.36× setting uses Nc = 24, N = 3, b = 24, while 1.8× uses Nc = 28, N = 6, b = 24.
D. Results on Wan 2.1
On Wan 2.1, STA maintains comparable performance across key evaluation metrics at the same sparsity level, while shorter videos reduce its end-to-end speedup.
- Wan 2.1: STA achieves comparable performance across key evaluation metrics on Wan 2.1 while preserving the same sparsity level.Wan 2.1 uses the same resolution but only 69 frames, resulting in reduced end-to-end speedup.
E. Results on Image Super-Resolution
The supplied passages identify an image-superresolution evaluation with FLUX and a separate sparse-kernel benchmark, but provide no numerical results for either.
- Image super-resolution: The image-superresolution evaluation uses FLUX on 1,000 captions randomly sampled from the COCO-2014 validation dataset.The passage identifies the evaluation setup but does not report its results.
- Sparse-kernel benchmark: Table 7 reports speedups for sparse attention kernels on an H100 GPU.The supplied passage does not provide the table's numerical comparisons.
F.2. Detailed VBench Results
Detailed VBench comparisons show STA's quality advantages over several sparse-attention baselines, while sparsity and fine-tuning affect different quality dimensions and visual outputs.
- VBench comparisons: STA surpasses Swin attention on Imaging Quality and Multiple Objects, while matching or exceeding CLEAR and Tiled NATTEN on these comparisons.The cited passage reports video-quality comparisons across detailed VBench dimensions.
- Effect of sparsity: Training-free STA shows lower temporal flickering and motion smoothness as sparsity increases, while Appearance Style, Color, and Spatial Relationships improve.The passage attributes the semantic-aligned improvements to the amplified role of text embeddings when spatial-temporal attention is sparsified.
- Effect of training: Trained STA substantially improves video-quality metrics over untrained STA while maintaining comparable semantic coherence.This indicates that training refines low-level visual fidelity without compromising text-video alignment.
- Qualitative comparisons: Fine-tuning STA-t-2.43x causes minor output-distribution shifts while preserving high video-generation quality.The qualitative comparisons also report that ∆-DiT videos are generally less sharp than outputs from original HunyuanVideo and STA.
- Qualitative comparisons: The qualitative examples include STA-tf-1.89x at 8 mins 21 s and STA-t-2.43x at 6 mins 29 s for two text prompts.The passages provide example labels and prompts but no additional VBench values.