Source-linked AI summary
FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
Hao Liu, Chenghuan Huang, Ye Huang, Zhiying Wen, Hao Liu, Mohan Zhang, Chen Li, Ziyang Ma, Jing Lyu, Jiangsu Du
TL;DR
Adaptive Top-p routing improves sparse-attention fidelity but creates rank-level stragglers under multi-GPU sequence parallelism. FVAttn repairs materialized masks with runtime head migration and slack-aware augmentation, achieving a 4.41× attention speedup over FlashAttention and consistently improving the quality–efficiency Pareto frontier across evaluated workloads.
Problem
Adaptive Top-p routing creates uneven head workloads and rank-level stragglers under sequence parallelism, limiting distributed sparse-attention efficiency.
Method
FVAttn combines adaptive Top-p routing with a Top-k floor, runtime P2P migration of heavy heads, and slack-aware sparse augmentation.
Results
4.41× attention speedup over FlashAttention was achieved on Wan2.2 I2V, while FVAttn consistently improved the quality–efficiency Pareto frontier across evaluated workloads.
Takeaways & Limitations
FVAttn improves distributed execution efficiency for dynamic sparse routing in multi-GPU video DiT inference.
Takeaways & Limitations
FVAttn primarily targets long spatiotemporal video generation, and its gains may decrease for short or weakly sparse workloads and weaker inter-GPU communication.
Abstract
from arXiv · showhide
Video Diffusion Transformers process long spatio-temporal sequences, making self-attention the main bottleneck in high-resolution video generation. Training-free sparse attention reduces this cost, but adaptive Top-$p$ routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse attention into a rank-level straggler problem. We present \method{}, a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism. \method{} uses Top-$p$ routing, a Top-$k$ safety floor, and video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path. Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks, while overlap hides scheduling and migration overhead behind existing computation. On step-distilled Wan2.2 I2V, \method{} reduces average load imbalance from 1.34 to 1.08 and delivers a $4.41\times$ attention speedup over FlashAttention, while achieving a $2.02$--$2.11\times$ DiT inference speedup with competitive video quality.
1 Introduction
Video DiTs face rapidly increasing inference latency as spatio-temporal sequences grow, while adaptive Top-p sparse attention creates rank-level stragglers under sequence parallelism. FVAttn repairs realized sparse masks at runtime through targeted head migration and slack-aware augmentation, reducing imbalance and accelerating attention and DiT inference.
- Motivation: Growing video resolution, frame count, and duration expand spatio-temporal token sequences and substantially increase Video DiT inference latency.Generating a 5-second 720p Wan2.2-14B I2V video on one NVIDIA H20 already takes approximately [value truncated in passage].
- Motivation: Top-p routing adapts retained-block counts to attention concentration and can preserve fidelity better than fixed Top-k at similar average compute.Concentrated heads retain fewer high-mass blocks, whereas flat or multimodal heads retain more interactions.
- Systems bottleneck: 97% maximum adjacent-step head-density variation and 44% rank-load variation make historical or offline layout decisions fragile in 4-step distilled video generation.Under Ulysses-style sequence parallelism, Top-p’s variable retained-block counts create skew and synchronized attention stragglers.
- Runtime repair: Slack-Aware Sparse Augmentation adds high-value blocks only to measured-slack ranks, converting synchronization idle time into coverage without extending global attention latency.RLB determines where existing sparse computation runs, while SASA spends residual slack using block-importance rankings.
- Evaluation: 1.34 to 1.08 average load-imbalance factor and 4.41× attention speedup over FlashAttention demonstrate FVAttn’s Wan2.2 I2V efficiency gains.The complete stack incurs only 0.7 ms of visible runtime overhead and improves the quality–efficiency Pareto frontier across evaluated workloads.
- Runtime repair: FVAttn migrates a small number of overloaded heads after sparse-mask materialization, using realized workloads and communication-budgeted P2P transfers to shorten the critical path.The workload skew is localized to a small subset of heavy heads, avoiding global repartitioning or a full parallel-layout redesign.
2 Background and Related Work
Video DiT sparse attention spans static patterns and dynamic routing, with FVAttn adopting dynamic block-sparse attention. Under Ulysses-style sequence parallelism, adaptive routing creates heterogeneous head workloads that translate into rank-level imbalance, motivating runtime load-balancing techniques.
- Sparse Attention: Training-free video-DiT sparse attention comprises static patterns with predetermined structures and dynamic routing with input-dependent masks; FVAttn uses dynamic block-sparse attention.Dynamic masks are constructed from current attention features.
- Sparse Attention: Top-k retains a fixed number of highest-scoring key blocks, whereas Top-p retains the smallest cumulative-importance set and produces more heterogeneous workloads.Top-p adapts computation to attention-distribution shape, allocating more to flat or multi-peaked distributions and less to concentrated ones.
- Sparse Attention: FVAttn combines Top-p routing with a Top-k safety floor to preserve a minimum computation budget, but heterogeneous per-head workloads create rank-level imbalance under fixed Ulysses placement.The safety floor addresses routing-estimation errors or overly aggressive sparsification.
- Sequence Parallelism: Ulysses-style sequence parallelism uses a sequence-to-head All-to-All so each GPU computes complete sequences for a subset of heads independently.Dense attention remains naturally balanced because heads have nearly identical compute, unlike dynamic sparse attention whose cost follows the current mask.
- Load Imbalance: 1.34 to 1.08 is the reported reduction in the load-imbalance factor, where synchronized attention latency is governed by the slowest rank.Values close to 1 indicate balanced execution; larger values indicate wasted waiting time on other GPUs.
- Related Work: Existing distributed sparse-attention systems address imbalance through mask-aware repartitioning, profile-based head placement, or pattern-specific parallel execution.Examples include db-SP, which repartitions workloads globally and reuses planning across steps, and S-HPLB, which uses offline sparsity profiles.
3 FVAttn Design
FVAttn is a runtime-scheduled sparse-attention pipeline that constructs video-aware adaptive masks, repairs realized rank imbalance through constrained head migration, and uses residual slack for additional high-value blocks. Nonblocking overlap reduces the visible cost of these runtime mechanisms while preserving their scheduling decisions.
- Pipeline: FVAttn executes adaptive mask construction, workload measurement, constrained P2P repair, slack-based augmentation, block-sparse attention, and reverse head restoration in sequence.This pipeline follows sequence-to-head All-to-All and concludes with reverse All-to-All.
- Sparse-routing frontend: Hilbert-curve block reordering, Top-p CDF thresholding, and a Top-k floor provide the video-aware sparse-routing frontend shared by RLB and SASA.The Hilbert curve places spatially adjacent tokens in nearby blocks, while Top-p adapts retained-block counts to each query block’s similarity distribution.
- Runtime Load Balancing: Top-p head-level sparsity variation amplifies into rank-level imbalance, making synchronized attention approximately governed by the maximum rank workload and creating straggler idle bubbles.FVAttn therefore schedules at head granularity and optimizes critical-path workload within a constrained migration space.
- Runtime Load Balancing: 1.34 to 1.08 average imbalance is achieved on Wan2.2 I2V by migrating only 20% of local heads, or one head per rank in the 8-GPU setting.Further increasing the migration budget yields much smaller gains, motivating lightweight P2P migration rather than global reshuffling.
- Slack-Aware Sparse Augmentation: SASA converts residual non-critical-rank slack into additional high-value key blocks without globally increasing the Top-p threshold.It follows routing’s importance order and risks creating a new straggler if augmentation exceeds residual slack.
- Overlap/Opt.: 1.87% visible overhead remains after Overlap/Opt., which issues scheduling, migration, and augmentation asynchronously and synchronizes only at true data dependencies.Overlap/Opt. also removes unnecessary CPU–GPU synchronization and streamlines runtime management and kernel launches.
4 Experiments
Experiments across three distilled Wan video workloads show that FVAttn improves the quality–efficiency trade-off, while ablations attribute gains to runtime load balancing, slack-aware augmentation, and overlap. Runtime repair reduces imbalance and critical-path latency without materially increasing visible overhead.
- End-to-end results: FVAttn improves the end-to-end quality–efficiency trade-off across Wan2.2 I2V, Wan2.2 Animate, and Wan2.1 T2V workloads.Experiments use the LightX2V 4-step distilled LoRA configuration and compare against FlashAttention and representative training-free sparse-attention baselines.
- End-to-end results: 19.10s vs. 19.92s: at I2V Top-p=0.95, FVAttn matches SpargeAttention latency while improving PSNR/SSIM/LPIPS/CLIP-Sim from 22.400/0.7728/0.1261/0.9880 to 23.801/0.8092/0.1045/0.9906.At Top-p=0.90, FVAttn also reduces DiT latency from 19.23s to 18.30s and improves VBench from 88.2% to 88.8%.
- Runtime ablations: 7.4%: adding RLB reduces Wan2.2 Animate DiT latency from 16.40s to 15.18s and raises speedup from 2.25× to 2.43× without changing PSNR/SSIM/LPIPS/CLIP-Sim.RLB changes head placement without altering selected QK blocks, and is 2.2% faster than db-SP at identical sparse-routing quality.
- Mechanism-level results: 1.34 to 1.08: RLB reduces final attention load imbalance and lowers latency to 41.32 ms, compared with 1.22 imbalance and 44.17 ms for db-SP.SASA further reduces effective imbalance from 1.08 to 1.01 while keeping attention latency essentially unchanged at 41.33 ms.
- Overhead analysis: ∼0.7 ms: Overlap/Opt. leaves only 1.87% visible critical-path overhead on the final 37.5 ms attention latency.CPU–GPU and computation–communication overlap hide balance planning, density exchange, and P2P migration behind existing computation; SASA contributes ∼0.2 ms visible cost.
5 Conclusion and Limitations
FVAttn is a training-free sparse-attention system that improves distributed execution efficiency for dynamic sparse routing in video DiT inference under multi-GPU sequence parallelism. Its benefits primarily apply to long spatiotemporal video sequences and depend on sufficient hardware communication and memory bandwidth.
- Conclusion: FVAttn improves distributed execution efficiency for dynamic sparse routing in video DiT inference under multi-GPU sequence parallelism.It is presented as a training-free sparse-attention system.
- Conclusion: Adaptive Top-p routing improves mask fidelity but creates head-level workload heterogeneity that becomes rank-level stragglers during multi-GPU execution.The routing adapts compute budgets to each attention head’s information demand.
- Limitations: FVAttn primarily targets long spatiotemporal video-generation sequences, where sparse-attention savings and load-balancing or slack-reuse benefits are more applicable.These benefits decrease for short sequences or workloads with weak attention sparsity, including many image-generation tasks.
- Limitations: FVAttn’s benefit depends on the hardware communication-to-computation ratio.The passage identifies H20-class servers with sufficient communication and memory bandwidth as a setting where lightweight P2P migration and overlap can be effective.