Source-linked AI summary
LVSA: Training-Free Sparse Attention for Long Video Diffusion
Gael Glorian, Ioannis Lamprou, Zhen Zhang, Yujie Yuan, Hongsheng Liu
TL;DR
Long-video diffusion faces quadratic attention costs and frozen or looping outputs beyond training horizons. LVSA introduces training-free sparse attention with rotating global anchors, delivering up to 3.33× speedup while improving extended-horizon quality.
Problem
Long-video diffusion inference faces quadratic attention costs, memory limits, and frozen or looping outputs beyond models’ training horizons.
Method
LVSA is a training-free, model-agnostic block-sparse attention algorithm using rotating sparse patterns and adaptive windows to reduce computation while preserving quality.
Results
LVSA-FI achieves up to 3.33× speedup over dense attention across three video diffusion models and enables HunyuanVideo 1.5 generation at a 2× horizon.
Takeaways & Limitations
LVSA provides compute and quality benefits across diverse video diffusion models, architectures, and hardware, including extended-horizon generation.
Takeaways & Limitations
The presented formulation uses a single global set of equidistant frames, with extensions to additional dedicated initial anchors left as straightforward generalization.
Abstract
from arXiv · showhide
Dense self-attention is the compute and quality bottleneck of long-video diffusion inference: cost grows quadratically with the sequence length, and beyond the training horizon the model converges to near-static output, that is, "frozen" repetitive video. State of the art approaches are either too costly, e.g., they require retraining, or fail to satisfy both performance and quality objectives in a scalable manner. To this end, we introduce Long Video Sparse Attention (LVSA), a training-free model-agnostic block-sparse attention for video diffusion transformers that combines a structured window pattern with rotating global anchors, thus removing the fixed-grid bias which causes long-range temporal artifacts. LVSA, combined with a FlashInfer kernel, reduces compute up to 3.17x on Wan 2.1 1.3B at a 6x horizon, 2.98x on Wan 2.1 14B at a 6x horizon, and 3.33x on HunyuanVideo 1.5 at a 1.5x horizon, compared to dense attention. Beyond reducing compute, LVSA enables HunyuanVideo 1.5 generation at a 2x horizon, which is otherwise out-of-memory on a single GPU. Moreover, LVSA provides speedups up to 2.41x compared to RIFLEx and 3.27x compared to UltraViCo on Wan 2.1 1.3B. To demonstrate applicability across diverse platforms, we apply LVSA on NPUs and achieve speedups up to 2.71x on Wan 2.2 A14B and 3.24x on Wan 2.1 1.3B compared to dense attention. To evaluate quality in a fair way, we introduce VQeval, a tool properly scoring loopy video failures, which instead are rewarded in state of the art evaluators like VBench-Long. LVSA is quality-neutral for generation at training horizon length and quality-positive at extended lengths.
1 Introduction
LVSA addresses the steep compute and quality challenges of long-video diffusion with training-free, model-agnostic sparse attention and a dedicated evaluation benchmark. Across multiple video diffusion transformers and hardware platforms, it delivers substantial speedups while improving long-horizon quality assessment.
- Motivation: Standard self-attention has quadratic compute, pushing 14-billion-parameter models near the 80 GB GPU memory limit for longer videos.Beyond training horizons, video quality also suffers from long-range temporal-repetition failures.
- Limitations of prior work: Existing training-free sparse patterns reduce attention cost, while quality-preserving extrapolation methods do not sufficiently reduce compute.Long-range temporal-repetition failures remain difficult to eliminate.
- LVSA: LVSA is a training-free, model-agnostic block-sparse attention algorithm with rotating sparse patterns and expanded adaptive window logic.Its design targets the tradeoff between compute and quality in long-video diffusion.
- Evaluation: VQeval is introduced to properly score loopy video failures, unlike the state-of-the-art evaluator VBench-Long.The benchmark is designed to address quality-evaluation shortcomings for long-horizon generation.
- Results: 3.17× speedup on Wan 2.1 1.3B, 2.98× on Wan 2.1 14B, and 3.33× on HunyuanVideo 1.5 are achieved with LVSA and FlashInfer on a single 80GB GPU.LVSA also significantly outscored dense attention on VQeval at a 6× horizon on both Wan models and enabled HunyuanVideo 1.5 generation at a 2× horizon.
- Results: 2.71× speedup on Wan 2.2 A14B and 3.24× on Wan 2.1 1.3B are achieved on NPUs at a 6× horizon with good video quality.These results demonstrate LVSA across hardware platforms.
2 Method
LVSA reduces long-video attention by restricting each query frame to local windows plus periodic global anchors, replacing dense O(N^2d) attention with a controlled sparse budget. Expanded windows maintain uniform coverage, while rotating global anchors remove the fixed-grid bias across denoising steps.
- Motivation: Dense attention costs O(N^2d) because every query frame attends to all T frames, motivating sparse per-frame attention.For one query, the cost is O(Nd) = O(TPd).
- Attention pattern: LVSA defines each query frame’s attended set as A(t) = G ∪ W(t), combining equidistant global frames with a local temporal window.Global frames occur every Tper frames, while W(t) spans up to 2W + 1 nearby frames with boundary clipping.
- Budget allocation: Expanded window bounds compensate for boundary clipping and global-window overlap, yielding |A(t)| = |G| + min(2W + 1, T −|G|) uniformly across frames.Across evaluated configurations, the adjustment loop averaged 1.21 iterations and took approximately 1.4 µs per frame on one CPU core.
- Rotating global frames: Rotating global frames shift G by s modulo Tper at denoising step s, ensuring every frame becomes a global anchor at least once per Tper-step cycle.This rotation addresses the persistent bias of fixed global frames and eliminates the fixed-grid bias, apart from negligible wrap-around revisits.
3 Experiments on GPU
GPU experiments evaluate LVSA across three video diffusion transformers, extended horizons, and complementary quality benchmarks. LVSA preserves quality at training length, improves quality beyond it, reduces wall time, and enables settings where dense attention exceeds GPU memory.
- Experimental setup: Experiments cover Wan 2.1 1.3B, Wan 2.1 14B, and HunyuanVideo 1.5 at 480 × 832 resolution across model-specific horizons.Wan is tested from 1× to 6× horizons, while HunyuanVideo 1.5 is tested from 0.5× to 2×.
- Experimental setup: All reported cells average five long descriptive prompts using seed 16, classifier-free-guidance scale 5.0, and each model’s default scheduler.Results in Sections 3.1 and 3.2 report mean ± standard deviation over the five-prompt set.
- Quality evaluation: VQeval complements VBench-Long by penalizing static or collapsed videos that VBench-Long can reward through inter-frame similarity.VQeval measures dynamic quality, loop quality, and text alignment, whereas VBench-Long includes subject consistency, temporal flickering, motion smoothness, background consistency, and imaging quality.
- Cross-model scaling: 3.17× speedup is achieved by LVSA-FI on Wan 2.1 1.3B at a 6× horizon, with corresponding gains of 2.98× on Wan 2.1 14B and 3.33× on HunyuanVideo 1.5.The comparison is against dense attention across the longest tested horizon for each model.
- Feasibility at the GPU memory ceiling: 60.3 GB / 60.4 GB peak memory lets LVSA run HunyuanVideo 1.5 at a 2× horizon on one 80GB GPU, while dense attention runs out of memory.The dense SDPA kernel attempts to allocate an additional 19.9 GB on top of a 74.0 GB resident process; LVSA uses SDPA / FlashInfer respectively.
- Quality across horizons: At 1× length, LVSA remains quality-neutral, with VQeval composite within ±1.0 and VBench-Long composite differences no larger than 0.014 versus dense.SDPA and FlashInfer produce equivalent quality at the training horizon, indicating the attention pattern rather than kernel choice determines output quality.
- Quality across horizons: +12.1 VQeval points is LVSA-FI’s advantage over dense on Wan 2.1 1.3B at 6×, while the advantage on Wan 2.1 14B reaches +12.2 at 6×.The advantage widens monotonically with horizon; at Wan 2.1 1.3B, LVSA-FI gains +4.7 at 2× and +11.6 at 4×.
- Comparison with extrapolation baselines: LVSA achieves the highest VQeval composite at every tested ratio against dense, RIFLEx, and UltraViCo on Wan 2.1 1.3B.Against dense, gains are +6.5, +11.2, and +9.9 at r = 2/3/4; against RIFLEx, +5.9, +11.2, and +8.7; against UltraViCo, +1.7, +1.9, and +3.5.
4 Experiments on NPU
LVSA’s port to vLLM-Omni demonstrates applicability across diverse hardware, delivering substantial NPU speedups and quality gains for long-horizon video inference.
- NPU experiments: 3.24× speedup at a 6× horizon on one NPU for Wan 2.1-1.3B, with a quality-positive result.The result uses 40-step inference over 481 frames at 720 × 1280 resolution with a standard NPU kernel.
5 Conclusion
LVSA is a training-free block-sparse attention method for long-video diffusion inference, demonstrating benefits across diverse models, architectures, and hardware in generation performance and quality.
- 5 Conclusion: LVSA is a training-free block-sparse attention method for long-video diffusion inference.The authors report benefits across diverse models, architectures, and hardware for generation performance and quality.
- 5 Conclusion: Future work may pursue further performance improvements and extend LVSA’s benefits to multi-scene video generation.