Source-linked AI summary
Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers
Bingxin Xing, Yu Zhang, Jiatong Ding, Lei Lu, Ping Wang
TL;DR
Repeated dense self-attention is a major DiT inference bottleneck, and the paper asks whether reusable high-mass structure can be identified offline. It introduces a train-free hybrid attention replacement and reports end-to-end speedups on T2V and T2I backbones with favorable quality–efficiency trade-offs.
Problem
Repeated dense self-attention is a major inference bottleneck as DiT token sequences grow and the operator runs across many denoising steps.
Source: §1 · p. 1Method
Sink-Sparse Hybrid Attention reuses offline-calibrated layer/head-specific sink masks and supplements them with a lightweight pooled-KV branch, without changing model weights or sampling schedules.
Source: §6 · p. 10Results
The method achieves 1.54× end-to-end speedup on Wan2.1-T2V-1.3B and 1.57× on Z-Image-Turbo, with favorable quality–efficiency trade-offs.
Source: §6 · p. 11Takeaways & Limitations
Reusable importance-aware sparse support provides a practical train-free acceleration path for the tested pretrained DiTs.
Source: §6 · p. 11Evaluation covers a limited set of backbones, requires backbone-specific offline calibration, and does not examine reliable mask transfer across backbones.
Source: §6 · p. 11
Abstract
from arXiv · showhide
Diffusion Transformers (DiTs) have become a common backbone for high-quality text-to-image and text-to-video generation, but repeated dense self-attention remains a major inference bottleneck as token sequences grow and the same operator is invoked across many denoising steps. Our study suggests that dense attention in pretrained DiTs contains reusable layer/head-specific high-mass structure that can be identified from a small offline calibration set and reused across future generations. This leads to Sink-Sparse Hybrid Attention, a train-free attention replacement that reuses static sink masks and supplements them with a lightweight pooled-KV branch, while leaving model weights and sampling schedules unchanged. On Wan2.1-T2V-1.3B, the method attains 1.54$\times$ end-to-end speedup while maintaining generation quality. On Z-Image-Turbo, it attains 1.57$\times$ end-to-end speedup with competitive GenEval2 performance. These results indicate that reusable importance-aware sparse support can provide an effective train-free acceleration path for both T2V and T2I DiTs.
1 Introduction
Repeated dense self-attention makes DiT inference costly as visual generation scales, while train-free sparsity must balance reusable structure against preserving useful context. Sink-Sparse Hybrid Attention addresses this with calibrated static sink masks and pooled-KV compensation, without retraining or changing sampling schedules.
1 Introduction
- Repeated global self-attention across long token sequences, layers, heads, and denoising steps remains a major inference bottleneck for video and high-resolution image DiTs.Optimized dense kernels reduce implementation overhead but do not change the quadratic interaction pattern.Source: §1 · p. 1
- Existing train-free sparse inference must balance cheap but rigid layouts against flexible approaches that incur online overhead and repeated pattern discovery.Fixed local windows may miss long-range dependencies, while adaptive selection can add runtime cost.Source: §1 · p. 2
- A small offline calibration set can identify layer/head-specific block supports that capture a large fraction of dense attention mass across prompts, seeds, and denoising steps.The paper calls these retained regions sink blocks, distinguishing them from causal LLM attention-sink tokens.Source: §1 · p. 2
- Sink-Sparse Hybrid Attention reuses calibrated static masks and adds pooled-KV compensation for residual global context, with normalization-aware fusion and no retraining or sampling-schedule changes.The sparse branch preserves recurrent high-mass interactions; the compensation branch uses full-resolution queries with pooled keys and values.Source: §1 · p. 2
- The framework is evaluated on both T2V and T2I diffusion transformers for quality–efficiency trade-offs.The contribution summary identifies both generation settings as validation targets.Source: §1 · pp. 2–3
2 Related Work
Prior work accelerates diffusion inference through reuse, attention approximations, sparse attention, and importance-aware structure. This paper focuses on train-free DiTs, calibrating model-specific block supports offline for reuse across denoising steps and future runs.
2.1 Efficient Inference for Diffusion Models
- Feature caching, attention reuse, and selective recomputation exploit repeated evaluations of nearby latent states to reduce diffusion-sampling computation.Examples include DeepCache, TGATE, FasterCache, ToCa, and AB-Cache [10] [11] [12] [25].Source: §2.1 · p. 3
2.2 Sparse Attention in Diffusion Transformers
- Train-free sparse attention methods for video DiTs exploit locality, sparse support, or adaptive search, while operator-oriented work develops reusable selection rules and efficient execution primitives.Other approaches incorporate sparsity through training, adaptation, or distillation [4].Source: §2.2 · p. 3
2.3 Importance-Aware and Structured Sparsity in Transformers
- Sink-Sparse Hybrid Attention studies reusable layer/head-specific block support in pretrained DiTs, with most pattern-discovery cost paid during offline calibration.This places the method between hand-designed sparse layouts and fully online pattern discovery.Source: §2.3 · p. 3
3 Motivation
Section 3 asks whether self-attention is a major runtime bottleneck and whether dense attention has reusable structure. It reports that attention takes 44%–51% of sampling time in the evaluated setting and that a static sink mask retains high attention mass across denoising steps.
3 Motivation
- A reusable static mask is motivated by targeting substantial latency while keeping retained support informative across prompts, seeds, and denoising phases.The section frames these as practical criteria for a train-free sparse operator.Source: §3 · p. 3
3.1 Self-Attention Is a Major Runtime Bottleneck
- 44%–51% of sampling time is self-attention in the evaluated long-sequence setting across denoising schedules.This profiling excludes model initialization and video writing.Source: Figure 1 · p. 4, §3.1 · p. 4
3.2 Stable High-Mass Attention Regions Across Inputs and Steps
- Stripe-like maps remain visually similar across prompts, and a static sink mask retains high dense-attention mass through the default 50-step trajectory.The representative mask has 38.8% mean retained block density (61.2% sparsity); a budget-matched random mask has much lower coverage.Source: Figure 2 · p. 4, §3.2 · pp. 4–5, Figure 3 · p. 5
4 Method
Sink-Sparse Hybrid Attention calibrates static layer- and head-specific masks offline, then reuses them during inference with an optional pooled-KV branch to approximate omitted context.
4.1 Notation and Overview
- The pipeline partitions tokens into blocks, calibrates static sink masks from dense observations, and reuses them during inference with either a sink-only or hybrid branch.Masks are estimated separately for each layer and head.Source: §4.1 · p. 5
4.2 Static Sink Mask Acquisition
- Calibration aggregates dense token attention into exact block-mass scores, averages observations, and retains each row’s smallest key-block set reaching coverage threshold τ.The resulting per-layer, per-head masks are reused across denoising steps; different backbones are calibrated independently.Source: §4.1 · p. 5, Figure 4 · p. 6, §4.2 · p. 6
4.3 Sink-Sparse Hybrid Inference
- The sink branch preserves recurrent high-mass interactions, while pooled-KV compensation uses full-resolution queries and pooled keys and values to supply omitted global context.The two branches are fused into the final attention output.Source: §4.3 · p. 6, Figure 4 · p. 6
- Row-wise fusion weights use the branches’ log-sum-exp normalizers with a pooled-branch support-size correction, allowing compensation to vary across query rows.Some rows are already well explained by sink support, while others benefit more from the pooled summary.Source: §4.3 · p. 7
4.4 Complexity and Deployment
- The hybrid cost combines sparse sink interactions with pooled-KV attention, which keeps query length N while reducing key/value length to about N/g_comp.Masks are calibrated once per backbone and reused without retraining, fine-tuning, or changing the denoising schedule.Source: §4.4 · p. 7
5 Experiments
The experiments evaluate Sink-Sparse Hybrid Attention on Wan2.1-T2V-1.3B and Z-Image-Turbo using model-specific offline masks and compare its runtime and quality with dense and sparse baselines. The results show faster generation on both tasks, with a tunable T2V quality–efficiency trade-off.
5.1 Experimental Setup
- The evaluation uses 50 denoising steps for 832×480, 81-frame T2V and eight steps for 2048×2048 T2I, with VBench Avg and Soft-TIFA GM as primary quality metrics.Masks are calibrated separately for each backbone from four prompts and two noise seeds, with statistics aggregated across denoising steps.Source: §5.1 · p. 7
5.2 Main Results
- At τ=0.85, Ours-Hybrid reaches 1.54× speedup on Wan2.1-T2V-1.3B, versus SpargeAttn’s 1.18×, with VBench Avg (8 dims) of 0.659 versus 0.665.It is faster than SVG1 (106.73s vs. 115.49s) and slightly higher in VBench Avg (8 dims) (0.659 vs. 0.656).Source: Table 1 · p. 8, §5.2 · p. 8
- On Z-Image-Turbo, Ours-Hybrid achieves 1.57× speedup and 15.21 Soft-TIFA GM, compared with SpargeAttn’s 1.47× and 15.07.Ours-Hybrid also scores 66.48 Soft-TIFA AM versus SpargeAttn’s 66.88.Source: Table 2 · p. 8, §5.2 · p. 8
5.3 Efficiency–Quality Trade-off
- At τ=0.60, Ours-Sink reaches 79.09% main-branch effective sparsity and 2.01× speedup, while Ours-Hybrid shifts the T2V quality–efficiency frontier upward at comparable sparsity.The pooled-KV branch adds modest latency while recovering a substantial part of the quality gap.Source: §5.3 · p. 8
Wan2.1-T2V-1.3B
The T2V analysis examines qualitative examples, calibration-set size, and the pooled-KV compensation branch. It finds that calibration quality gains largely saturate at moderate budgets, while compensation helps most under aggressive sparsification.
Wan2.1-T2V-1.3B
- Figure 5 presents three synchronized video frames for Wan2.1 and image examples ordered Dense, Ours-Hybrid, then SpargeAttn for Z-Image-Turbo.The video frames use the same noise seed across methods.Source: Figure 5 · p. 9, Wan2.1-T2V-1.3B · p. 9
5.4 Calibration Size
- Calibration-set quality gains largely saturate by 4×4: VBench Avg (8 dims) rises from 0.645 at 2×2 to 0.647 at 4×2 and 0.650 at 4×4, while 8×4 yields 0.649.Runtime stays nearly unchanged across budgets, and the authors identify 4×2 or 4×4 as practical trade-offs.Source: Table 3 · p. 9, §5.4 · pp. 9–10
5.3 Efficiency–Quality Trade-off
- In T2V, τ=0.70 is the stronger speed-oriented point and τ=0.85 the quality peak; between them is a knee, with further sparsification below causing more visible quality loss and denser masks above yielding smaller incremental gains.Source: Wan2.1-T2V-1.3B · p. 9
5.5 Effect of Hybrid Compensation
- Hybrid compensation improves VBench Avg (8 dims) over Ours-Sink at every tested threshold, with gains of +0.071 at τ=0.60, +0.063 at τ=0.65, and +0.066 at τ=0.70.The quality gap narrows as the static sink mask becomes denser, but the branch remains beneficial.Source: Table 4 · p. 10, §5.5 · p. 10
6 Conclusion
The paper concludes that Sink-Sparse Hybrid Attention combines reusable calibrated sink structure with pooled-KV compensation for pretrained DiTs. Its tests report speedups on two backbones, while the authors note limits in backbone coverage and mask adaptation.
6 Conclusion
- The train-free framework identifies reusable layer/head-specific sink structure from a small offline calibration set, preserves it with a static sparse branch, and adds a lightweight pooled-KV branch.Source: §6 · p. 10
- The method achieves 1.54× end-to-end speedup on Wan2.1-T2V-1.3B and 1.57× on Z-Image-Turbo; the authors say pooled-KV restores coarse omitted context at modest cost.They describe reusable importance-aware sparse support as a practical acceleration path for pretrained DiTs.Source: §6 · p. 11
- Evaluation covers a limited set of backbones, requires backbone-specific offline calibration, and does not test whether masks transfer reliably across backbones.The authors also note that fixed masks may adapt less well to distribution shifts or unusual prompts, and aggressive sparsity may lose rare or strongly input-dependent dependencies.Source: §6 · p. 11