PaperlayerView exact revision on arXiv

Source-linked AI summary

DraftAttention2: Fast Video Diffusion with Low-Resolution-Guided Mixed-Precision Attention

Rui Ding, Haopeng Li, Weize Ma, Yufa Zhou, Yitong Li, Xiaoling Zhou, Jiashuo Cao, Liyang Li, Hua Geng, Jiuxiang Gu, Jun Lin, Enze Xie, Xuan Shen

arXiv:2609.32628v1cs.CVcs.AI

TL;DR

Attention over spatiotemporal tokens makes longer, higher-resolution video generation computationally expensive. DraftAttention2 uses a low-resolution draft to jointly allocate attention sparsity and precision, and reports better quality and dense-output fidelity than competing methods at matched speedups.

  • Problem

    Attention over spatiotemporal tokens scales quadratically with sequence length and is repeated at every denoising step, making longer and higher-resolution video generation computationally expensive.

    Source: §1 · p. 1

  • Method

    DraftAttention2 uses complementary average- and max-pooled queries and keys to rank attention blocks, jointly assigning their precision or skipping them under configurable budgets.

    Source: §1 · p. 2

  • Results

    At matched speedups, DraftAttention2 achieves better generation quality and dense-output fidelity than SpargeAttention on HunyuanVideo and MiniMax-H3, and better generation quality with comparable acceleration than SageAttention3.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    Recovering selected sparse interactions with low-bit computation can substantially tighten the output-error bound relative to the original DraftAttention under matched allocations.

    Source: §5 · p. 9

    The reconstruction model uses original BF16 inputs as its reference, leaves the 16-bit path unquantized, and treats ordinary floating-point roundoff separately.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Video generation has broad applications in content creation and entertainment. Diffusion transformers have advanced the quality of generated videos, but attention over spatiotemporal tokens becomes increasingly expensive as video resolution and duration increase. We present DraftAttention2, a training-free framework that uses the low-resolution draft attention map to jointly select attention blocks and assign their numerical precision. Specifically, spatial 2D average- and max-pooled queries and keys capture complementary regional statistics to estimate block importance, and a shared ranking assigns higher precision to important blocks, lower precision to less important retained blocks, and skips the rest under configurable budgets. Our analysis separates sparsification error from attention-weighted quantization error, establishing when recovering skipped interactions with low-bit computation tightens the output-error bound. This analysis motivates retaining more interactions at low precision while reserving higher precision for blocks with larger attention mass. To translate these fine-grained assignments into practical speedups, we further develop fused operand preparation and a single attention kernel with precision-specific phases, sharing data movement, softmax statistics, and output accumulation across precisions. Experiments demonstrate that our method achieves a superior quality-efficiency trade-off over existing efficient video generation methods. Notably, its advantage is particularly pronounced for few-step video diffusion, where jointly combining sparsity with 4- and 8-bit mixed-precision computation substantially improves generation quality while retaining significant acceleration. Code is available at https://github.com/anemoi-project/anemoi

1 INTRODUCTION

Attention over spatiotemporal tokens is a major cost in video diffusion, growing quadratically with sequence length and recurring at every denoising step. DraftAttention2 jointly allocates sparsity and precision using low-resolution attention guidance, with theoretical analysis and fused execution supporting its quality–efficiency trade-off.

1 INTRODUCTION

  • Attention over spatiotemporal tokens scales quadratically with sequence length and is repeated at every denoising step, making it a major DiT runtime bottleneck.The introduction identifies longer videos and higher resolutions as settings where computational expense remains substantial.Source: §1 · p. 1
  • At similar measured latency, mixed 4/8-bit attention achieves lower output error than single-precision sparse and dense allocations.The joint allocation replaces some skipped interactions with low-precision computation while reserving higher precision for important blocks.Source: §1 · p. 2
  • A low-resolution draft combines average- and max-pooled queries and keys to rank blocks, assigning higher precision to important blocks, lower precision to other retained blocks, and skipping the rest.The ranking adapts retained blocks and precision assignments across query regions under configurable budgets.Source: §1 · p. 2, Figure 2 · p. 2
  • The analysis separates sparsification error from attention-weighted quantization error, finding that low-bit computation for moderately important interactions can tighten the output-error bound.It also motivates assigning higher precision to interactions with larger attention mass.Source: §1 · p. 2
  • Fused operand preparation and precision-specific phases in one attention kernel share data movement, softmax statistics, and output accumulation, reducing mixed-precision execution overhead.The phases avoid separate per-precision outputs and an additional merging kernel.Source: §1 · p. 2
  • DraftAttention2 shows a particularly strong advantage over existing efficient attention methods in few-step video diffusion, where approximation errors are less likely to be corrected.The experiments report competitive generation quality with acceleration; a small higher-precision block budget improves similarity to dense-attention outputs while preserving most of the speed benefit of low-precision computation.Source: §1 · pp. 2–3

2 RELATED WORK

Prior video-generation methods exploit attention sparsity through profiling, pooled block selection, mask reuse, or clustered-token routing. Low-bit and mixed-precision methods quantize attention computations or assign precision to selected regions; DraftAttention2 jointly selects blocks and assigns precision using one pooled-importance ranking.

2 RELATED WORK

  • Sparse video-generation methods use online pattern profiling, pooled-query/key block selection, mask reuse, or clustered-token top-p routing (Shen et al., 2025).Sparse VideoGen profiles patterns per head; DraftAttention uses spatial average pooling; AdaSpa reuses masks between designated search steps; Sparse VideoGen2 clusters and reorders tokens.Source: §2 · p. 3
  • Low-bit methods quantize attention matrix multiplications or combine quantization with block selection and softmax-aware filtering (Zhang et al., 2025a;b;c; Zhang et al., 2025d).Block-level mixed-precision approaches use diagonal regions or offline-calibrated per-head boundaries.Source: §2 · p. 3
  • ThriftAttention assigns FP16 to top-k key blocks and FP4 to remaining unmasked blocks, while DraftAttention2 jointly selects blocks and assigns multiple precisions using a shared per-head ranking (Sharratt, 2026).DraftAttention2 allows retained-block counts and precision allocations to vary across query regions under a common per-head budget.Source: §2 · p. 3

3 METHODOLOGY

DraftAttention2 uses a low-resolution draft to allocate both which attention blocks are retained and the precision used for each. Its analysis motivates recovering some skipped interactions at low precision, while its execution design shares preparation and attention computation across precision paths.

3.1 DRAFT-GUIDED PRECISION ALLOCATION

  • Average- and max-pooled regional queries and keys produce draft importance scores that rank region pairs within each head for 16-bit, 8-bit, 4-bit, or skipped computation.Head-level quotas determine how many blocks receive each precision, while the total retained fraction is T/R^2.Source: §3.1 · p. 4

3.2 QUANTIZATION AND ATTENTION IMPORTANCE

  • At similar measured latency, the illustrative mixed 4/8-bit allocation reduces relative output error to 3.132%.The comparison contrasts omitted interactions in sparse 16-bit attention with greater quantization error in dense 4-bit attention.Source: §3.2 · p. 5
  • The error analysis separates sparsification error from attention-weighted quantization error, showing when low-bit recovery of skipped interactions tightens the output-error bound.For matched allocations, the bound improves when the added quantization term is smaller than the reduction in sparsification error.Source: §3.2 · p. 5
  • With retained support and precision quotas fixed, assigning higher precision to region pairs with larger attention mass minimizes the assignment-dependent error bound when ε8 ≤ ε4.The router uses draft scores as low-cost proxies because exact attention masses require full-resolution attention.Source: §3.2 · pp. 5–6

3.3 HARDWARE-EFFICIENT EXECUTION

  • Spatial regions are packed into contiguous blocks, while fused Q/K and V preparation reuses operand movement to produce draft descriptors and prepared attention operands.The partition uses less than one block of total padding per frame and is reused across layers and denoising steps.Source: §3.3 · p. 6, Figure 3 · p. 6
  • A single kernel processes retained blocks in precision-specific phases, sharing online softmax statistics and output accumulation across phases.Precision transitions depend on the active formats rather than the number of retained blocks, and output normalization occurs once after all retained blocks.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

DraftAttention2 is evaluated on HunyuanVideo and MiniMax-H3, including few-step MiniMax-H3 variants, against dense attention and efficient-attention baselines. Results report quality, fidelity, and speed trade-offs, while ablations examine pooling and precision allocation.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers HunyuanVideo-13B and MiniMax-H3, including 4-step and 8-step FL2VA Turbo LoRA variants; quality uses VBench, dense-output similarity uses PSNR, SSIM, and LPIPS, and latency is measured on RTX 5090.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • For 4- and 8-step MiniMax-H3 generation, DraftAttention2 improves quality and dense-output fidelity over sparse-only baselines at comparable or higher acceleration.At speedups comparable to SpargeAttention, mixed 4/8-bit allocation further improves dense-output fidelity.Source: Table 2 · p. 8, §4.2 · p. 8
  • At comparable speedups, DraftAttention2 improves generation quality and dense-output fidelity over SpargeAttention on both HunyuanVideo and MiniMax-H3, and improves generation quality over SageAttention3.Sparse configurations reach up to 2.38× speedup on HunyuanVideo and 2.33× on MiniMax-H3, exceeding sparse-only baselines.Source: §4.2 · pp. 7–8
  • In the 4- and 8-step visual comparisons, DraftAttention2 preserves the rail junction and cat’s appearance, producing results close to the dense baseline.Figure 4 shows 8-step examples at the top and 4-step examples at the bottom; red boxes mark structural details.Source: §4.2 · p. 8, Figure 4 · p. 9

4.3 ABLATION STUDY

  • Adding MaxPool to average pooling improves the error–latency trade-off for 8-bit and mixed 8/16-bit attention, with video-generation gains at fixed sparsity and precision budgets.The reported improvements extend to both 16-bit and mixed 8/16-bit attention.Source: §4.3 · p. 9, Table 3 · p. 9
  • Across latency budgets, assigning a small fraction of computation to higher precision lowers attention output error versus single-precision alternatives, including beyond pure 8-bit attention’s zero-sparsity endpoint.Reducing sparsity alone does not remove quantization error when every block uses low precision.Source: §4.3 · p. 9, Figure 5 · p. 9

5 CONCLUSION

DraftAttention2 jointly allocates sparsity and precision using a low-resolution attention draft, with fused execution for its fine-grained assignments. Experiments show a strong quality–efficiency trade-off, especially in few-step video diffusion.

5 CONCLUSION

  • DraftAttention2 uses a low-resolution draft to jointly allocate sparsity and numerical precision in video diffusion, with one importance ranking guiding block selection and precision assignment.Fused operand preparation and shared accumulation across precision phases reduce execution overhead.Source: §5 · p. 9
  • The analysis separates sparsification error from attention-weighted quantization error and shows that low-bit recovery of selected skipped interactions can tighten the output-error bound versus DraftAttention under matched allocations.Source: §5 · p. 9
  • On HunyuanVideo and MiniMax-H3, DraftAttention2 achieves a strong quality–efficiency trade-off, with particular advantages in few-step generation while retaining significant acceleration.Mixed-precision recovery improves generation quality and dense-output fidelity in the few-step setting.Source: §5 · p. 9

A ADDITIONAL EXPERIMENTAL DETAILS

The appendix specifies generation settings and metrics, and describes how attention error and latency are measured.

A ADDITIONAL EXPERIMENTAL DETAILS

  • The standard models retain dense attention for their first 10 denoising steps, while few-step variants retain it for the first quarter of steps.The few-step setup uses FL2VA Turbo 4-step and 8-step LoRAs; latency covers the complete DiT trajectory, including preparation and retained dense attention.Source: A ADDITIONAL EXPERIMENTAL DETAILS · p. 13
  • Video quality is scored with eight VBench dimensions and their arithmetic mean, while dense-output fidelity uses PSNR, SSIM, and LPIPS on matched prompts and random seeds.Frames are normalized to [0, 1] for PSNR and SSIM; LPIPS uses the AlexNet backbone.Source: A ADDITIONAL EXPERIMENTAL DETAILS · p. 13
  • Attention relative L2 error is measured against dense output, and latency is the median of three complete-operator measurements.Latency-budget comparisons select the lowest measured error among configurations within the budget; the single-precision reference includes 4-, 8-, and 16-bit paths.Source: A ADDITIONAL EXPERIMENTAL DETAILS · p. 13

B QUANTIZATION ANALYSIS AND PROOFS

The analysis fixes the router-selected support and separates interaction removal from quantization approximation on retained interactions.

B QUANTIZATION ANALYSIS AND PROOFS

B.1 RECONSTRUCTION-ERROR MODEL

The appendix specifies a reconstruction-error model for retained attention blocks, then derives operand and weight-error quantities under shared normalization.

B.1 RECONSTRUCTION-ERROR MODEL

  • B.1 models additional low-bit quantization relative to stored BF16 reference tensors, while accounting for ordinary floating-point effects separately.The reference-precision path has no additional low-bit quantization cost; the ordering of low-bit error envelopes is an assumption, not a consequence of bit width alone.Source: §B.1 · p. 15

B.2 RECONSTRUCTED OPERANDS AND LOGIT ERROR

  • B.2 bounds reconstructed logit and value errors per block, using actual residuals for floating-point formats and a step-based bound for unclipped uniform quantization.The INT8 scalar error is at most half the recovery step; the vector bound scales with dimension.Source: §B.2 · pp. 15–16

B.3 WEIGHT QUANTIZATION UNDER SHARED NORMALIZATION

  • B.3 defines reconstructed exponential weights at a common row reference, with the denominator using pre-conversion weights and the numerator using quantized weights.Online row-maximum changes rescale the numerator and denominator together, preserving the block-relative error under those common rescalings.Source: §B.3 · p. 16

B.4 PROOF OF THE ATTENTION-WEIGHTED ERROR BOUND

The proof bounds the retained-support output error by accounting for logit, value, and exponential-weight quantization, then separates implementation roundoff.

B.4 PROOF OF THE ATTENTION-WEIGHTED ERROR BOUND

  • The retained-support output error has separate contributions from logit, value, and exponential-weight quantization, including normalization changes caused by perturbed logits.The block-dependent bound follows by combining the per-block result with the format-level error envelope.Source: §B.4 · p. 17
  • Ordinary floating-point rounding and reference-path format conversion are accounted for separately from the low-bit quantization bound.Omitting a 16-bit quantization term does not assume exact hardware arithmetic.Source: §B.4 · p. 17

B.5 PROOF OF THE V1-TO-V2 ERROR-BOUND COMPARISON

The comparison derives a condition under which recovering skipped attention mass with low-bit computation tightens the output-error bound, within a matched-support operator comparison.

B.5 PROOF OF THE V1-TO-V2 ERROR-BOUND COMPARISON

  • The proof compares v1, which skips additional mass τ, with v2, which recovers that mass in low-bit paths while retaining the shared 16-bit support.The v2 bound is subtracted from the v1 bound to obtain the condition for a tighter bound.Source: §B.5 · p. 17
  • The bound comparison is for reconstructed operators with matched 16-bit supports and isolates low-bit recovery from changes in draft scoring or support selection.It compares dense attention masses, not block fractions; equal sparsity ratios alone do not establish the result.Source: §B.5 · p. 17
  • The comparison concerns output-error bounds; measured latency and relative output error are evaluated separately in operator experiments.Arithmetic roundoff can be added separately.Source: §B.5 · p. 17

B.6 PROOF OF IMPORTANCE-GUIDED PRECISION ALLOCATION

The proof frames precision assignment as minimizing an attention-mass-weighted error objective at fixed support and quotas, then establishes an ordering by importance.

B.6 PROOF OF IMPORTANCE-GUIDED PRECISION ALLOCATION

  • With retained support and always-retained precisions fixed, support error and those fixed interactions contribute constants to the assignment objective.The retained-pair weights are independent of precision under the fixed-support setup.Source: §B.6 · p. 18
  • Assigning a higher error-cost format to a larger attention-mass weight incurs excess cost compared with the reversed assignment.Exchanging inverted assignments preserves quotas and cannot increase the objective, yielding the stated ordering.Source: §B.6 · p. 18
  • The theorem applies to the error bound at fixed support and quotas; the implemented router approximates mass ordering with A_ab rather than evaluating weights or error envelopes online.The pooling mixture controls this proxy and is outside the quantization-error decomposition.Source: §B.6 · p. 18

C IMPLEMENTATION DETAILS

The implementation combines block-wise precision assignment with fused operand preparation and a phased attention kernel that shares softmax state and output accumulation. It also specifies routing, spatial packing, quantization formats, and additional visual comparisons.

C IMPLEMENTATION DETAILS

  • Operand preparation is fused, and precision-specific kernel phases share data movement, softmax statistics, and output accumulation.The phases use a common online-softmax state.Source: C IMPLEMENTATION DETAILS · p. 18

C.1 ROUTING AND PRECISION FORMATS

  • Within each head, ranked video-region pairs receive higher precision, lower precision, or are skipped under configurable budgets.Region-pair assignments apply to their physical attention blocks; prefix interactions are handled outside the video-region budget.Source: §C.1 · p. 18

C.2 FUSED QUANTIZATION AND REORDERING

  • Frames are partitioned into connected regions and packed into contiguous blocks; minimizing partition perimeter preserves more spatial neighbors within pooled regions.Valid-token counts exclude padding, and an inverse index map restores output order.Source: §C.2 · pp. 18–19

C.3 QUANTIZATION FORMATS AND SCALES

  • The kernel distinguishes operand and accumulator formats, keeps shared row maxima, exponential sums, and output state in FP32, and supports NVFP4, INT8, and FP16 paths.NVFP4 uses E2M1 values with E4M3 local scales; INT8 uses INT8 for QKᵀ and FP8 for probability–value multiplication.Source: §C.1 · p. 18, §C.3 · p. 19, Table 6 · p. 19

C.4 PHASED MIXED-PRECISION EXECUTION

  • NVFP4, INT8, and FP16 phases extend one online-softmax state, rescaling the accumulated numerator and denominator when the running row maximum changes.The final output divides the numerator by the shared denominator.Source: §C.4 · p. 20

D ADDITIONAL VISUAL COMPARISONS

Related papers

Loading 2609.32628v1…