Source-linked AI summary
Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification
Haopeng Li, Yitong Li, Junsong Chen, Tian Ye, Haozhe Liu, Jincheng Yu, Duomin Wang, Ruihua Zhang, Zeke Xie, Enze Xie, Song Han
TL;DR
Long token sequences make self-attention a dominant video-generation bottleneck, while existing sparse methods incur routing overhead and lose information by dropping unselected blocks. Sol-Attn unifies threshold routing, sparse computation, and proxy-score-based correction in one online-softmax pass, achieving 2.1×–3.0× end-to-end speedups across video tasks while preserving generation quality.
Problem
Long token sequences make self-attention a dominant inference bottleneck, while existing block-sparse methods rely on costly offline routing and keep-or-drop sparsification.
Method
Sol-Attn unifies query-dependent threshold routing, sparse computation, and proxy-score reuse for approximate correction within a single online-softmax pass.
Results
2.1×–3.0× end-to-end speedups across video tasks were achieved while preserving generation quality and advancing the quality–efficiency frontier.
Takeaways & Limitations
The unified streaming design narrows the accuracy gap between sparse and dense attention with little additional overhead across visual-generation tasks.
Takeaways & Limitations
Sol-Attn currently supports only forward inference, and evaluation is limited to bidirectional diffusion-based visual generation rather than autoregressive video generation.
Abstract
from arXiv · showhide
Diffusion transformers are essential for high-fidelity video generation, but long token sequences make attention a dominant inference bottleneck. Training-free dynamic sparse attention alleviates this bottleneck by computing only selected key-value blocks, yet existing methods struggle to sparsify attention both efficiently and accurately for two reasons: (1) Rigid, unpredictable, and costly routing: selecting a fixed fraction of top-ranked blocks by proxy score imposes fixed budgets, whereas retaining blocks to reach a target cumulative proxy probability mass yields dynamic but potentially imbalanced budgets; both incur non-negligible overhead from computing and materializing proxy scores. (2) Lossy keep-or-drop sparsification: unselected blocks are discarded entirely, degrading accuracy under aggressive sparsity. These limitations motivate cheaper dynamic-budget routing while limiting accuracy degradation. In this paper, we introduce training-free Sol-Attn (Sparsifying online attention), which unifies dynamic routing, sparse computation, and approximation correction in a single online-softmax pass, achieving a better accuracy-efficiency trade-off in sparse attention. The core of Sol-Attn is on-the-fly block thresholding with proxy-score reuse, which selects critical blocks by comparing block proxy scores against a threshold during online softmax. This design enables dynamic yet controllable block budgets without materializing the proxy map, while directly reusing the proxy scores of unselected blocks to approximate their contribution. Experiments across image and video generation tasks show that Sol-Attn advances the quality-efficiency frontier of training-free sparse attention, delivering 2.1 times and 2.3 times end-to-end speedups for video generation and editing, respectively, while preserving visual quality.
1. Introduction
Sol-Attn addresses the inference bottleneck of long-token diffusion-transformer video generation by combining threshold-based dynamic sparsification, online softmax, and proxy-score reuse in one streaming operator. Across image and video tasks, it improves the quality–efficiency frontier while preserving generation quality and achieving substantial end-to-end speedups.
- Motivation: Longer, higher-resolution video generation creates quadratic self-attention costs that dominate inference, motivating training-free dynamic sparse attention for pretrained models.Existing block-sparse methods remain limited by costly, rigid routing and lossy keep-or-drop sparsification.
- Sol-Attn: Sol-Attn fuses threshold-based dynamic sparsification into online softmax and reuses proxy scores to approximate unselected blocks within a single kernel.This design targets a better accuracy–efficiency trade-off without changing model weights.
- Query-dependent thresholding: Query-dependent thresholds select blocks under dynamic yet controllable budgets using lightweight score statistics, without materializing the full proxy map.The thresholds are estimated from statistics such as mean and standard deviation.
- On-the-fly sparsification: On-chip threshold comparisons route blocks tile by tile during online softmax, eliminating storage of full proxy maps and routing indices in HBM.Scores are consumed as generated, and selected blocks enter a nested sparse-attention loop over original key-value tiles.
- Results: 2.1×–3.0× end-to-end speedups across video tasks demonstrate Sol-Attn’s efficiency gains, reaching 5.1× with complementary Sol-Engine optimizations while preserving generation quality.Experiments cover image generation, video generation, editing, and refinement.
2. Preliminaries
FlashAttention’s tiled online softmax avoids materializing the full attention matrix and its quadratic memory footprint, but still evaluates all query-key interactions. Block-sparse attention skips masked key-value tiles at block granularity to reduce computation, while standard proxy-score routing trades fixed budgets for difficult-to-control dynamic budgets.
- Attention and Online Softmax: FlashAttention avoids materializing the full attention matrix through tiled on-chip online softmax, reducing I/O overhead and eliminating the O(L^2) memory footprint.It still evaluates all query-key interactions, so its compute cost remains quadratic.
- Block-Sparse Attention: Block-sparse attention reduces O(L^2) computational complexity by computing only critical key-value pairs for each query.Block granularity lets implementations skip masked key-value tiles entirely, converting sparsity into compute savings.
- Block-Sparse Attention: Standard routing partitions sequences into query and key-value blocks and uses mean-pooled block proxy scores to construct the sparse mask.The proxy score for each query-key block pair is derived from the means of the corresponding query and key blocks.
- Block-Sparse Attention: Top-k enforces a rigid fixed budget, whereas top-p yields a dynamic but hard-to-control budget; both require traversing proxy scores.Figure 2 contrasts these strategies with Sol-Attn’s threshold derived from proxy-logit mean μ, standard deviation σ, and standardized offset β.
3. Methodology
Sol-Attn combines query-dependent threshold routing, sparse attention, and approximate correction in one online-softmax pass. Its Gaussian-calibrated thresholds provide controllable average sparsity with query-dependent budgets, while proxy-score reuse preserves skipped-block contributions without materializing the proxy map.
- Dynamic threshold routing: Sol-Attn derives query-dependent thresholds from proxy-score statistics and uses a shared standardized cutoff β to control model-level mean density.The threshold maps each query block’s mean and standard deviation to the raw proxy-score scale, avoiding fixed per-query budgets.
- Dynamic threshold routing: Thresholds are computed in O(Ld + Nd^2) time with O(d^2) auxiliary storage, without materializing the N × N proxy-score map.Mean and variance are obtained directly from pooled-key first and second moments.
- On-the-fly sparsification: Chunk-wise streaming thresholding selects blocks as proxy scores are produced and recovers the same block set as full-row thresholding.Dense chunk scans determine selected blocks, which are then traversed sparsely in a nested-loop attention structure without intermediate routing tensors.
- Approximate correction: Unselected blocks retain approximate softmax numerator and denominator contributions through zeroth-order proxy-score reuse, while selected blocks receive exact computation.The approximation uses pooled keys and summed values for skipped blocks, limiting accuracy loss under sparsity.
- Unified execution: A shared token-to-block score tile simultaneously determines routing and approximate correction, fusing both with exact sparse attention in one online-softmax pass.The kernel maintains one online-softmax state while streaming pooled-key/value chunks and loading original key-value blocks only for routed indices.
4. Experiments
Experiments across video, image, consumer-GPU, and integrated inference settings show that Sol-Attn improves the quality-efficiency frontier while preserving similarity to dense attention. Ablations attribute these gains to scalable kernel execution, efficient threshold routing, stable dynamic budgets, approximate correction, and unified execution.
- Efficient On-the-Fly Sparsification: 11.5× and 32.7× faster routing than top-k and top-p, respectively, is achieved by consuming proxy scores on chip instead of materializing and globally traversing the proxy map.Tile-local routing state also keeps Sol-Attn near dense attention in processor memory, while SVG2 requires about 8× more memory on both workloads.
- Text-to-Video Generation: 2.12× end-to-end speedup is achieved for text-to-video generation at matched sparsity, with state-of-the-art efficiency across the evaluated models.The evaluation spans three models and compares VBench quality and similarity to dense-attention outputs.
- Text-to-Image Generation: 1.56× speedup is achieved on native 2K text-to-image generation, while outperforming other sparse methods in Qwen-Image-Bench scores and dense-reference similarity.The task uses Ideogram 4 and evaluates moderate sequence lengths.
- Full-Stack Inference: 3.48× and 5.08× end-to-end speedups are achieved on Wan2.1-14B and HunyuanVideo by integrating Sol-Attn with diffusion-step caching and kernel fusion in Sol-Engine.The combined system is evaluated on NVIDIA B200 GPUs.
- Ablations: Approximate correction lowers relative ℓ2 error and preserves higher cosine similarity than exact-only sparse attention under identical selected blocks as sparsity increases.Query-dependent thresholding also produces tightly concentrated density distributions around a 15% mean density while retaining dynamic block budgets.
5. Related Work
Prior work reduces attention cost through structured or content-based block selection, learned sparse architectures, and correction-based approximations. Dynamic routing commonly relies on proxy-score thresholds or top-k/top-p masks, while correction methods approximate omitted blocks’ contributions.
- Block Sparse Attention: Block sparse attention selects key-value blocks to reduce quadratic attention cost, using structured layouts, content-based grouping, or learned sparse architectures.Recent approaches also combine compression, selection, and local attention within native sparse architectures.
- Routing Strategies: Dynamic routing methods select informative blocks with top-k or top-p proxy scores, often adding confidence, thresholding, or combined masks.SpargeAttn adds confidence and online-softmax-aware thresholds, while SpargeAttn2 unions top-k and top-p masks to address complementary failure modes.
- Sparse Attention with Correction: Correction-based sparse attention approximates contributions from unselected blocks alongside exact sparse computation.PISA uses block-wise Taylor expansion, SVG-EAR uses centroid-based linear compensation with error-aware routing, and BA-Att uses covariance-compensated block approximation.
6. Conclusion
Sol-Attn unifies threshold routing and approximate correction in one online-softmax pass, narrowing the accuracy gap between sparse and dense attention with little added overhead. Its current implementation and scope remain limited, motivating future kernel and application extensions.
- Contribution: Sol-Attn combines threshold routing and approximate correction in a single online-softmax pass without materializing a proxy map.The design treats routing as part of streaming sparse attention and approximates unselected blocks rather than discarding them.
- Contribution: The unified streaming design substantially narrows the accuracy gap between sparse and dense attention with little additional overhead.
- Limitations and Future Work: Sol-Attn’s B200 kernel does not fully exploit Blackwell’s performance potential, supports only forward inference, and has been evaluated only on bidirectional diffusion-based visual generation.The evaluation does not cover autoregressive video generation, motivating further kernel optimization and broader application support.
A. Additional Experimental Details
The experiments use dense-attention warm-ups before applying Sol-Attn, with schedules varying across video-generation models. Cross-attention remains dense in some models, while text key-value tokens are retained as sinks in exact attention branches.
- Generation Configurations: Wan2.1-14B and HunyuanVideo-13B use 50 denoising steps, applying dense attention for the first 10 steps and Sol-Attn thereafter.The passage specifies this schedule for both models.
- Generation Configurations: Bernini-14B uses 40 denoising steps with an eight-step dense warm-up before Sol-Attn.Its warm-up policy differs from the 50-step schedules of Wan2.1-14B and HunyuanVideo-13B.
- Generation Configurations: Wan2.1 and Bernini keep cross-attention dense, while HunyuanVideo retains text key-value tokens as sinks in the exact branch.The passage distinguishes cross-attention handling across these model families.
- Generation Configurations: Ideogram 4 uses eight dense steps in a 48-step schedule and retains text key-value tokens in the exact branch.This configuration parallels HunyuanVideo’s token-retention policy.
B. Derivations and Error Analysis
The section derives the exact threshold-variance expression from proxy-score variance and introduces a diagonal covariance approximation for efficient estimation. It also characterizes approximation error, showing second-order denominator behavior and identifying conditions behind accuracy and residual error.
- Threshold variance derivation: The exact threshold variance follows by expanding squared proxy scores and collecting pooled-key second-moment terms.The derivation substitutes the squared-score expression into the variance calculation and recovers Eq. (5).
- Diagonal threshold estimator: The diagonal estimator replaces dense pooled-key covariance with element-wise second moments and an O(d) projection per query block.It retains only the diagonal v_K := diag(Cov[¯K]) while providing both exact and approximate implementations.
- Approximation error: The error analysis defines centered score deviations and compares exact with zeroth-order block numerator and denominator contributions.For one query row, the analysis uses the maximum centered score spread η to characterize approximation error.
- Approximation error: The denominator error is second order in centered score spread, while numerator error additionally depends on alignment with values.These bounds explain accuracy on smooth tail blocks and residual error on heterogeneous blocks.
- Approximation error: The derivation assumes dense, noncausal keys within each block, while row-dependent semantic or causal masks require centering and multiplicity over valid keys.This assumption matches the evaluated diffusion models.
C. Additional Qualitative Results
Additional qualitative results compare dense attention with Sol-Attn across multiple text-to-video prompts and models, and show conditional-generation and editing cases. The examples span transportation, planetary, portrait, cockpit, desert, and object-transformation scenarios.
- Text-to-video comparisons: Additional prompts test detailed cinematic portrait and cockpit scenes involving lighting, facial expression, flight motion, and tropical landscapes.The portrait prompt specifies sunlight moving across a woman’s face, while the cockpit prompt describes a seaplane pilot banking above an archipelago.
- Text-to-video comparisons: Figure 11 presents dense-attention and Sol-Attn text-to-video comparisons across different models.The accompanying prompts include a motorcycle on a coastal highway, a train on tracks, a Mars rover, and a steam train on a mountainside.
- Video editing: Editing cases transform a porcelain fox into storm-filled glass and a red raincoat into moss and flowers while preserving specified motion, composition, and scene elements.The edits respectively preserve the fox’s motion and gallery setup, and the woman’s walk cycle, architecture, framing, and daylight.
- Text-to-video comparisons: A further text-to-video prompt depicts a stationary viewpoint across a dry lakebed with a black sports car, mountain ranges, and a blue sky.The scene also specifies desert texture, distant ridgelines, thin clouds, and crisp midday shadows.
- Conditional-generation comparisons: Figure 12 compares conditional generation on Bernini and SANA-WM using source, dense, edited, low-resolution, and refined video outputs.Bernini rows show source, dense, and Sol-Attn-edited videos; SANA-WM rows show stage-1 and both refined outputs.