MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers

1 Introduction

• Longer video and 3D token sequences make quadratic attention a major inference bottleneck, but aggressive sparsity can degrade generation quality and fidelity.Block-level sparsity also entangles approximate selection scores with coarse block-layout constraints, making the sources of the quality gap hard to separate.Source: §1 · p. 2
• Matched-density oracle comparisons identify structural binding, selection, and discarded-tail errors as distinct sources of sparse-attention quality loss.These findings motivate finer-grained selection, accurate attention scoring, and compensation for discarded contributions.Source: §1 · p. 2
• Efficient token-level selection must address costly exact scoring, irregular KV gathering, GPU tile requirements, and the loss of natural discarded-region summaries.Unconstrained query clustering can produce variable-sized groups that waste computation when padded to kernel tiles.Source: §1 · p. 2
• MC-Sparse caches tile-aligned query groups, exact-mass KV selections, and dense–sparse output residuals at anchor steps for reuse during subsequent denoising steps.The framework is training-free and uses fast PDDP grouping, a two-pass exact selector, and a token-sparse attention kernel.Source: §1 · p. 2
• Across evaluated video models, MC-Sparse improves dense-output fidelity and denoising speed over sparse-attention baselines; it reaches a 1.80× denoising speedup over dense attention on Minimax-H3-Base and a 2.32× denoising speedup on HY3D-Internal.Both reported speedups are at 15% attention density; HY3D-Internal has negligible quality degradation.Source: §1 · pp. 2–3

2.1 Long-Sequence Diffusion Transformers

• Video and 3D generation can require tens to hundreds of thousands of tokens, making quadratic full attention a major contributor to inference latency.The passage identifies increasing generation resolution as a driver of longer sequences.Source: §2.1 · p. 3

2.2 Efficient Attention

• Trainable efficient-attention work spans linear, sparse, and hybrid methods, including SANA, VMoBA, and SLA.Other named examples include SANA-Video, MHLA, VSA, SpargeAttention2, and SLA2.Source: §2.2 · p. 3
• Training-free approaches reduce evaluated token interactions using structured or content-dependent block selection, approximate compensation, and cross-step reuse.Semantic clustering can create variable-sized groups that mismatch kernel tiles and reduce execution efficiency; other methods reuse attention-related information along the denoising trajectory.Source: §2.2 · pp. 3–4

3 Preliminaries

• Dense attention computes each head’s output by applying softmax to scaled query–key scores and multiplying the resulting weights by values.The softmax is taken over keys, and Aij denotes the attention weight from query i to key j.Source: §3 · p. 4
• Block-sparse attention masks selected query–key interactions and retains entire KV blocks, sharing one selection among queries in a block.The block size is typically aligned with the GPU computation tile.Source: §3 · p. 4
• Vanilla BSA scores query-block/KV-block pairs using mean-pooled query and key representations, then retains the highest-scoring KV blocks at a given attention density.The selector operates at each diffusion step.Source: §3 · p. 4

4 Method

• Block-sparse attention loses fidelity through structural binding, selection error, and discarded tail error, isolated by comparing matched-density oracle selectors.The oracles select KV blocks, shared tokens per query group, or individual tokens per query using exact dense-attention probabilities.Source: §4 · p. 4

4.1 Mitigating Structural Binding Error

• Finer KV granularity and grouping more similar queries improve fidelity by reducing KV binding and query binding, respectively.MC-Sparse selects individual KV tokens across block boundaries and uses equal-size, tile-aligned groups of similar queries.Source: §4.1 · p. 5

4.2 Mitigating Selection Error

• Exact selection chooses the K KV tokens with the greatest cumulative dense-attention mass for a query group, avoiding pooled scores that can miss high-mass tokens.The exact selections are computed at anchor steps and reused, since nearby-step attention patterns are stable and reused exact selections outperform recomputed mean-pooled selections.Source: §4.2 · p. 5

4.3 Mitigating Discarded Tail Error

• Even per-query oracle selection leaves an output gap because discarded tokens make nonzero contributions; MC-Sparse reuses a dense–sparse residual that varies less across denoising steps than attention outputs.Exact selection further reduces residual variation relative to mean-pooled selection.Source: §4.3 · p. 6

4.4 Pipeline and Efficient Implementation

• MC-Sparse reuses query groups, exact KV indices, and dense–sparse output residuals across denoising steps, computing them at anchor steps and using them at subsequent reuse steps.At reuse steps, current Q, K, and V are recomputed.Source: §4 · p. 4, §4.4 · p. 6
• The implementation includes a two-pass exact-selection kernel, a CuTeDSL token-sparse kernel, Fast PDDP query-index propagation, and CPU cache offloading with prefetching.Prefetching overlaps transfers with computation; query-index propagation reduces peak memory usage.Source: §4.4 · pp. 6–7

5.1 Experimental Setup

• Evaluations cover three video models and HY3D-Internal, using task-specific prompts or image–prompt pairs and metrics for dense-output fidelity, image quality, geometry, consistency, and denoising speed.Video metrics include PSNR, SSIM, LPIPS, VBench ImgQual and BgCons; 3D metrics include CD, volumetric IoU, F1, Uni3D-I and ULIP3D-I.Source: §5.1 · p. 7

5.2 Main Results

• 28.44 dB PSNR at 25% density and 1.61× speedup beats Sol-Attn’s 23.66 dB and 1.59× at 32.4% density on Minimax-H3-Base.MC-Sparse-Flash reaches 1.80× speedup at 15% density with 27.30 dB PSNR; on Hunyuan and Wan, MC-Sparse has the best PSNR and SSIM and lowest LPIPS among evaluated sparse methods. Figure 7 reports videos visually closer to dense-attention outputs.Source: §5.2 · pp. 7–8, Table 1 · p. 8, Figure 7 · p. 8
• At 15% attention density, MC-Sparse reaches 2.32× denoising speedup and reduces CD from PISA’s 0.976 to 0.177 on HY3D-Internal.It achieves the best geometric fidelity among evaluated sparse methods, with input-image consistency comparable to dense attention; Figure 8 reports better preservation of fine details and surface continuity.Source: §5.2 · pp. 8–9, Table 2 · p. 9, Figure 8 · p. 9

5.3 Ablation Study

• Adding exact KV selection with cross-step reuse, token-level granularity, query grouping, and residual compensation successively improves PSNR, SSIM, and LPIPS across evaluated densities.The cumulative ablation is on Wan2.1-1.3B-T2V at 480p.Source: §5.3 · p. 9, Table 3 · p. 10
• At matched realized density, token-level KV selection improves recall and output accuracy across grouping strategies; Fast PDDP outperforms k-means while avoiding alignment inflation.Table 4 identifies alignment inflation as the increase in executed interactions over nominal sparse interactions after tile alignment.Source: §5.3 · p. 9, Table 4 · p. 10
• Residual compensation gives the largest cumulative-ablation gain, with substantially greater benefit alongside MC-Sparse selection and grouping than with vanilla BSA.PISA’s smaller additional gains are consistent with its native block-statistics compensation also addressing discarded tail contributions.Source: §5.3 · p. 10, Table 5 · p. 10

5.4 System Efficiency

• 96%–99% of ideal 1/s speedup is achieved by MC-Sparse’s token-sparse kernel, versus 80%–81% for SVG2 and 82%–87% for PISA; Fast PDDP cuts grouping cost 3.96× at 480p and 6.14× at 720p.Kernel comparisons are at matched KV-token densities on Wan2.1-14B-T2V at 720p and exclude selection and grouping.Source: §5.4 · pp. 10–11, Table 6 · p. 11, Table 7 · p. 11
• Measured auxiliary overhead is equivalent to ∆s ≈5%–6%; in both evaluated models, prefetching one layer ahead hides cached-residual and KV-index transfer latency.The overhead includes anchor-step evaluation and selection, query grouping, and other auxiliary operations.Source: §5.4 · p. 11, Table 8 · p. 11

• Oracle comparisons show that sparse-attention fidelity loss includes structural binding and discarded tail contributions, not only selection accuracy.Source: §6 · p. 11
• MC-Sparse combines token-level KV selection and tile-aligned query grouping with temporal reuse and residual compensation.Source: §6 · p. 11
• Experiments on video and 3D asset generation report improved fidelity to dense-attention outputs and reduced denoising latency against evaluated sparse-attention baselines.Source: §6 · p. 11

A.1 Evaluation Protocol

• Video evaluation randomly selects 50 samples per task; Minimax-H3-Base uses prompt enhancement and generates 345-frame videos, approximately 14.4 seconds long.Source: §A.1 · p. 15
• Analysis and ablations use Wan2.1-1.3B-T2V at 480p, with metrics sampled uniformly across diffusion steps and transformer layers outside dense warm-up regions.Source: §A.1 · p. 15
• MC-Sparse uses each baseline’s dense warm-up configuration, with anchor steps also listed; baseline settings follow SVG-EAR where applicable, and methods do not specially treat condition tokens.The protocol also gives a Sol-Attn diagonal-estimator setting and applies the same configuration to HY3D-Internal.Source: §A.1 · p. 15, Table 9 · p. 15, Table 10 · p. 15
• SpargeAttn runtime is omitted because its INT8 or FP8 matrix multiplications are not directly comparable with other methods’ BF16 measurements; HY3D-Internal PISA speedup is estimated as an upper bound.PISA’s provided kernel cannot run efficiently on HY3D-Internal without modification, so the estimate uses same-density Sol-Attn kernel latency.Source: §A.1 · pp. 15–16

A.2 Implementation Details

• Wan 2.1 and HunyuanVideo use optimized operators, with FlashAttention-3 for dense warm-up; Minimax-H3-Base uses Diffusers operators and FA3 for dense attention and warm-up.Source: §A.2 · p. 16
• Wan 2.1, HunyuanVideo, and HY3D-Internal run on one Hopper GPU; Minimax-H3-Base uses eight, with seven denoiser GPUs and Ulysses sequence parallelism.Source: §A.2 · p. 16
• Exact selection uses two passes: the first returns dense outputs and row-wise log-sum-exp values, and the second recovers probabilities for group-wise aggregation and top-K selection.The procedure avoids materializing the dense attention matrix.Source: §A.2 · p. 16
• The CuTeDSL token-sparse kernel asynchronously gathers selected key and value rows while consumer warpgroups compute attention, overlapping the next gather with current computation.Source: §A.2 · p. 16

• The section provides qualitative comparisons with full attention across video-generation models.Source: Figure 9 · p. 17