PaperlayerView exact revision on arXiv

Source-linked AI summary

Triage: Skip, Skim, or Refine Attention Tiles in Video Diffusion Transformers

Yeonjae Kim, Omin Kwon, Seong Hoon Seo, Jae W. Lee

arXiv:2610.35706v1cs.LGcs.CV

TL;DR

Existing sparse and low-bit methods use one precision for computed tiles, although approximation error varies widely across tiles and attention is a major quadratic-cost component of video DiT inference. TRIAGE uses block statistics to predict tile error under skip, NVFP4 skim, and NVFP4 residual-refine tiers, upgrading a tile when its predicted marginal gain justifies the extra compute. Across four video DiTs in five settings, TRIAGE achieves a better speed–fidelity trade-off than all evaluated baselines.

  • Problem

    Existing sparse and low-bit methods use one precision for computed tiles, although approximation error varies widely across tiles and attention is a major quadratic-cost component of video DiT inference.

    Source: §1 · p. 1

  • Method

    TRIAGE uses block statistics to predict tile error under skip, NVFP4 skim, and NVFP4 residual-refine tiers, upgrading a tile when its predicted marginal gain justifies the extra compute.

    Source: §1 · p. 2

  • Results

    Across four video DiTs in five settings, TRIAGE achieves a better speed–fidelity trade-off than all evaluated baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across three video DiT families, choosing precision tile by tile gives a better speed–fidelity trade-off than existing methods without retraining or modifying the model.

    Source: §9 · p. 11

    End-to-end gains are bounded because dense steps and work outside attention remain unchanged; LTX-2.5 gains less because only its three stage-2 steps run sparsely.

    Source: §7.2 · p. 9

Abstract

from arXiv · show

Attention over tens of thousands of visual tokens dominates the inference of video diffusion transformers (DiTs). Existing methods reduce this cost through sparsity, low precision, or both, but use a single precision for all kept tiles. We observe that the output error introduced by approximation varies widely across tiles. Many tiles can be skipped with little error, others must be computed but gain little beyond low precision, and a few retain substantial error even at low precision. We present Triage, a training-free attention method that assigns each tile one of three approximation tiers: skipping it by replacing it with its block means, skimming it in NVFP4, or refining it with NVFP4 residual correction. A lightweight router predicts each tile's error under each tier from block statistics and moves a tile to a more accurate tier only when the predicted error reduction justifies the extra compute, with a single price setting this trade-off. A fused kernel executes all three tiers in a single pass on the FP4 tensor cores of RTX Blackwell GPUs. Across Wan2.2, HunyuanVideo, and LTX-2.5, Triage accelerates end-to-end generation by up to 2.32x over dense FlashAttention, and in each of five settings it runs faster than every baseline while reaching 0.3-3.0 dB higher PSNR than the highest-PSNR baseline.

1 INTRODUCTION

Video DiT attention is costly at scale, while existing sparse and low-bit methods apply a single precision to all computed tiles despite large variation in approximation error. TRIAGE chooses among three per-tile tiers, achieving a better speed–fidelity trade-off than evaluated baselines.

1 INTRODUCTION

  • Attention over tens of thousands of visual tokens has quadratic cost and accounts for a large, growing share of inference time.The share grows as models move to higher resolutions and longer videos.Source: §1 · p. 1
  • Sparse, low-bit, and combined methods favor different attention distributions, but use one precision for all kept tiles.Sparse attention tends to work best on sharp rows, low-bit on diffuse rows, and their combination on intermediate rows.Source: §1 · p. 1, Figure 1 · p. 1
  • Approximation error varies widely across tiles, so uniform precision can spend excess compute on some tiles while leaving avoidable error on others.The paper motivates choosing extra computation by its marginal error reduction per unit of compute.Source: §1 · p. 2
  • TRIAGE assigns each tile to skip with block means, skim in NVFP4, or refine with NVFP4 residual correction, upgrading only when predicted marginal gain justifies the compute.A single price parameter controls the speed–fidelity trade-off.Source: §1 · p. 2
  • A fused kernel runs all three tiers on FP4 tensor cores, with skipped tiles represented as virtual tiles of block means and selected tiles refined by residual products.The method is training-free.Source: §1 · p. 2
  • Across four video DiTs and five settings, TRIAGE achieves a better speed–fidelity trade-off than all evaluated baselines.Across Wan2.2, HunyuanVideo, and LTX-2.5, generation runs up to 2.32× faster than FlashAttention.Source: §1 · p. 2

2 BACKGROUND

Video diffusion transformers apply 3D full attention across visual tokens, making attention cost quadratic in token count. Block-sparse methods select tiles to compute, while low-bit attention reduces the cost of each computed tile through quantization.

2.1 Video Diffusion Transformers

  • Video diffusion transformers iteratively denoise compressed video latents divided into visual-token patches, and 3D full attention makes attention cost quadratic in total token count.Each token attends to every other token across all frames.Source: §2.1 · p. 2

2.2 Block-Sparse Attention

  • Block-sparse attention processes query–key tiles row by row and selects tiles based on estimated attention mass, with skipped tiles optionally represented by block means.A row is the set of tiles attended to by one query block; dynamic selection and token clustering are training-free techniques discussed in the background.Source: §2.2 · p. 3

2.3 Low-Bit Attention and NVFP4

  • NVFP4 quantizes values in 4-bit E2M1 groups with shared E4M3 scales, and its low-bit tensor-core operations reduce the cost of computed tiles.The described smoothed-score computation uses a full-precision correction term, quantized probabilities and values, and FP32 accumulation and online softmax.Source: §2.3 · p. 3

3 MOTIVATION

The motivation analysis finds that the best allocation depends both on a row’s attention distribution and on variation among tiles within that row. This motivates selecting approximation precision per tile by marginal error reduction.

3.1 Different Approximations Across Rows

  • At equal modeled cost, sparse attention is most accurate on sharp rows, combined attention on intermediate rows, and low-bit attention on diffuse rows.The analysis compares allocations at 25% of modeled dense BF16 cost using dense-run attention outputs from Wan2.2-TI2V-5B and HunyuanVideo.Source: §3.1 · pp. 3–4, §3 · p. 3, Figure 3 · p. 4
  • Approximating a single tile produces row-output error that grows roughly with its attention mass, with NVFP4 error about one-fifth to one-quarter of block-mean error.The comparison keeps all other tiles exact.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Different Precisions Within a Row

  • Within a row, heavier tiles can yield more error reduction from extra precision even though increasing precision costs about the same across equal-sized tiles.The paper defines marginal gain as output error removed per unit of additional compute.Source: §3.2 · p. 4
  • At equal compute, TRIAGE reduces summed modeled error by matching each tile’s computation to its mass, unlike allocations that use one precision for kept tiles.A row may contain heavy tiles with substantial NVFP4 error alongside moderate tiles that are costly to upgrade uniformly.Source: §3.2 · p. 4, Figure 4 · p. 4

4 OVERVIEW

TRIAGE plans per-tile approximation tiers from clustered blocks and block statistics, then executes those tiers together in one online-softmax pass. Its design combines lower-error approximations with a single execution path.

4 OVERVIEW

  • Planning clusters similar queries and keys into fixed-size blocks, collects block statistics, and uses a router to select each tile’s tier without computing all attention scores.The clustering tree reduces within-tile variation and the statistics estimate tile mass and approximation error.Source: §4 · p. 4
  • Planning outputs a tier map, block means, and quantized operands for execution, while statistics collection and routing are fused with smoothing and quantization.This keeps planning overhead small.Source: §4 · p. 5
  • Execution combines skip, skim, and refine in one online softmax: skipped tiles contribute cluster means, skimmed tiles use NVFP4 products, and refined tiles add NVFP4 residual products.The residual refinement nearly matches BF16 accuracy.Source: §4 · p. 5

5 PLANNING: CHOOSING TIERS

TRIAGE plans per-tile approximation tiers using tile-aligned clustering and a router that prices marginal error reduction against compute. It estimates errors from block statistics and fuses routing-related preprocessing with smoothing and quantization.

5.1 Tile-Aligned Clustering

  • TRIAGE uses per-head divisive clustering to group similar tokens while controlling block sizes compatible with fixed-size tiles.Trees are rebuilt every five sparse steps; query leaves hold 128 tokens and key clusters hold 64, with two key clusters supplying statistics for each 128-key tile.Source: §5.1 · p. 5

5.2 Router

  • The router selects each tile’s tier by marginal error reduction per compute, using a price π as the smallest gain that justifies extra compute.A first-order approximation makes row error a sum of per-tile errors, allowing independent tier choices; dividing by tile count keeps π comparable across sequence lengths.Source: §5.2 · p. 6
  • Under Gaussian scores and uniformly random query directions, block statistics estimate skipping error; skimming is modeled as a fixed fraction α of that error, and refinement error is set to zero.The skipping estimate combines lost softmax mass and value shift, weighted by the square root of the head’s relative output energy.Source: §5.2 · p. 6
  • With α = 0.22 and crf = 3 fixed across models and settings, π is the only tuned parameter; each tile moves through skim and refine when each marginal gain meets π.The tier choice corresponds to where a line of slope −π touches the tile’s error–compute curve; the figure’s numbers are illustrative.Source: §5.2 · pp. 6–7, Figure 6 · p. 6

5.3 Preprocessing Kernels

  • TRIAGE fuses statistics collection, mass estimation, and tier assignment with smoothing and quantization in a few operand-grouped Triton passes.The preprocessing reuses BF16 smoothing scores and computes other routing scores from NVFP4 operands on FP4 tensor cores.Source: §5.3 · p. 7

6 EXECUTION: RUNNING ALL TIERS IN ONE PASS

TRIAGE executes skip, skim, and refine tiers in one attention-kernel pass, representing skipped tiles with virtual cluster-mean tiles and refining selected tiles with NVFP4 residual products. The kernel balances uneven query-block costs and is designed for RTX Blackwell GPUs.

6 EXECUTION: RUNNING ALL TIERS IN ONE PASS

  • TRIAGE runs all three tiers in one kernel on FP4 tensor cores, adding NVFP4 residual products only for refined tiles.The kernel is implemented for sm 120 RTX Blackwell GPUs and extends a warp-specialized NVFP4 attention kernel.Source: §6 · p. 7, §6.3 · p. 7

6.1 Skipping with Virtual Tiles

  • Skipped tiles are represented by virtual cluster-mean keys and values, with a log-count correction so the means represent all replaced keys without double-counting kept tiles.The virtual tiles join the same online softmax as the real keys.Source: §6.1 · p. 7

6.2 Refining Tiles on FP4 Tensor Cores

  • Refinement quantizes operands and their residuals; refined tiles add products containing one residual and omit the second-order product of two residuals.The paper states that three FP4 products take less time than one BF16 product at peak rates, avoiding an extra kernel, output reload, and merge pass.Source: §6.2 · p. 7

6.3 Attention Kernel

  • A persistent kernel assigns query blocks in largest-first snake order to spread costly blocks across cooperative thread arrays.Block cost is estimated from kept and refined tiles; the combined kernels form a drop-in backend for transformer self-attention.Source: §6.3 · p. 7

7 EVALUATION

Across five settings, TRIAGE improves the end-to-end speed–fidelity trade-off over the evaluated baselines, while attention-call tests and ablations examine its performance and design choices. The evaluation also compares router-score factors and NVFP4 refinement with a separate BF16 pass.

7.1 Setup

  • Evaluation covers four video DiTs in five settings on single Blackwell GPUs, using 50 prompts or image–prompt pairs and dense FlashAttention-2 as reference.Methods replace only visual-token self-attention; the first 20% of steps and first transformer block run densely, and comparisons use training-free sparse, low-bit, and combined baselines.Source: §7.1 · p. 8

7.2 End-to-End Performance

  • Across all five settings, one TRIAGE price is faster end to end than every baseline and closer to dense on PSNR, SSIM, and LPIPS.At π = 0.15, PSNR is 0.8–3.0 dB above SVG-EAR; at π = 0.3, speedup is 1.38–2.32× and PSNR is 0.4–8.0 dB above the fastest baselines.Source: Table 1 · p. 8, §7.2 · pp. 8–9

7.3 Attention Accuracy and Speed

  • At 4.5% and 4.0% error, TRIAGE runs 5.7× and 6.8× faster than FlashAttention, while SpargeAttn needs 1.8× and 2.0× more time at the same error.The comparisons pool captured attention calls from Wan2.2-TI2V-5B and HunyuanVideo 540p, including each method’s preprocessing time.Source: §7.3 · p. 9

7.4 Ablations

  • Removing refinement leaves error floors of 9.6% and 6.9%, while removing the clustering tree raises them to 12.9% and 7.3% on Wan2.2-TI2V-5B and HunyuanVideo.Refinement lowers the floors to 2.0% and 1.3%; at equal error, it makes TRIAGE 1.5× and 1.9× faster, while removing the tree makes the no-refinement variant 1.2–1.3× slower on HunyuanVideo.Source: §7.4 · p. 9

7.5 Router Score

  • Each router-score factor helps: omitting Jt + Rt raises error by 18% in 95% of calls, removing head weighting raises it by 7%, and mass corrections reduce it by 3%.The ablation holds the number of kept and refined tiles equal to the full score in every captured call.Source: §7.5 · pp. 9–10, Table 2 · p. 10

7.6 Refinement in NVFP4 versus BF16

  • NVFP4 residual refinement matches separate BF16 refinement within 0.1 points at both prices, while the BF16 pass slows the kernel by 34–43%.The kernel timing excludes preprocessing.Source: §7.6 · p. 10, Table 3 · p. 10

8 RELATED WORK

Related work reduces attention cost through sparsity, mixed precision, and acceleration outside attention. TRIAGE targets within-step attention cost and can be combined with methods that reduce other model work.

Sparse attention in LLMs

  • LLM sparse attention selects blocks per head and input during prefill, while decoding methods select KV-cache entries to reduce memory traffic rather than computation.MInference and XAttention select dynamically; NSA and DeepSeek Sparse Attention learn selection during training.Source: §8 · p. 10

Mixed-precision attention in LLMs

  • LLM mixed-precision methods compute selected blocks in FP16 and the rest in FP4 or INT8, using mean-pooled query–key scores or static templates.The passage identifies ThriftAttention and TileMix, respectively.Source: §8 · p. 10

Orthogonal acceleration

  • Post-training quantization, step distillation, and feature caching reduce work outside attention and can be combined with TRIAGE’s within-step attention acceleration.TurboDiffusion is cited as combining step distillation with low-bit and sparse attention.Source: §8 · p. 10

9 CONCLUSION

TRIAGE allocates attention precision tile by tile, and its conclusion reports a better speed–fidelity trade-off than existing methods across three video DiT families. The appendices describe tile-aligned clustering and the first-order basis for treating tile errors separately.

9 CONCLUSION

  • Across three video DiT families, TRIAGE achieves a better speed–fidelity trade-off than existing methods without retraining or modifying the model.The conclusion links this choice to making video generation practical as attention takes a growing share of generation with longer and higher-resolution videos.Source: §9 · p. 11

A THE CLUSTERING TREE

  • The tile-aligned PDDP tree projects tokens onto principal components, then repeatedly sorts and splits nodes along principal directions at whole-block boundaries.With fixed r, T, and b, the loop costs O((Nr^2 + N log N) log(N/b)), in addition to the initial projection.Source: A THE CLUSTERING TREE · p. 14

B ROUTER DERIVATIONS

  • The appendix derives tile-error decoupling and explains the approximations underlying the router score.The exact output is written as U/Z, with Z the summed softmax normalizer and U the summed weighted-value output.Source: B ROUTER DERIVATIONS · p. 14, §B.1 · p. 14

B.1 Decoupling the Tiles

  • The router’s separable objective uses the sum of single-tile relative errors as a first-order row-error bound when normalizer perturbations are small.For finite changes the sum can fall below the stated bound, and error magnitudes need not add because vector changes can cancel.Source: §B.1 · p. 14

B.2 The Error of Skipping a Tile

The skipping model estimates tile error from lost attention mass and the shift caused by replacing attention-weighted values with cluster means. The router score uses block-level moments and mass estimates to make these approximations inexpensive.

B.2 The Error of Skipping a Tile

  • Replacing a tile by cluster means causes both lost softmax weight and a value shift, because mean values weight tokens equally rather than by attention.The router approximates skipping error as tile mass multiplied by the estimated lost-mass and value-shift terms; its unit-scale substitutions and query-block aggregation mean this is neither exact nor a certified upper bound.Source: §B.2 · p. 15
  • The mass estimates use Gaussian-score corrections for within-block query variation and are approximate, not guaranteed lower bounds.One estimate scores a mean query against keys with a query-side variance correction; another scores queries against cluster means and corrects for estimated lost mass.Source: §B.2 · pp. 15–16

B.3 The Router Score

  • The router estimates key variation and value shifts from within-cluster statistics, while between-cluster variation is retained by the mean representatives.Its listed inputs include query-block statistics, cluster key and value moments, and tile-level aggregation across the two key clusters.Source: §B.3 · p. 16
  • The router converts head-relative errors to a common layer scale using a head weight derived from output norms.The head weight uses the previous call’s mean squared output norm, with a current-call fallback at the first call.Source: §B.3 · p. 16

C.1 End-to-End Details

In the evaluated end-to-end settings, TRIAGE is faster than sparse baselines despite computing a greater share of tiles, while retaining fidelity to dense videos. The results also show how its tier allocation varies with attention entropy.

C.1 End-to-End Details

  • 2.1–2.6× faster than SVG-EAR and 1.06–1.61× faster than Sol-Attn, TRIAGE computes more tiles at π = 0.15 while keeping each computed tile in NVFP4.SVG-EAR and Sol-Attn compute 17–26% and 22–23% of tiles, respectively; keeping more tiles supports TRIAGE’s fidelity while low-precision computation keeps it fast.Source: §C.1 · p. 16
  • 47%, 84%, 71%, 85%, and 80% of dense-step attention time is consumed by attention in the five evaluated settings, limiting the end-to-end gains available from attention acceleration.At π = 0.15, TRIAGE reaches 88%, 75%, and 75% of the first three theoretical bounds; at π = 0.3, it reaches 76% and 81% of the last two.Source: §C.1 · p. 16
  • Every method stays within 0.008 of dense in the VBench mean across five dimensions, and TRIAGE stays within 0.003.PSNR, SSIM, and LPIPS separate methods more clearly than VBench in these comparisons.Source: §C.1 · p. 16

C.2 Tiers Across Attention Entropy

  • 61–84% of tiles are skipped and 3–12% refined in every entropy quintile on Wan2.2-TI2V-5B, while refined tiles hold 95% of mass in sharp rows.In diffuse rows, at most 3% of tiles are refined and mass is split between skimmed tiles and many low-mass skipped tiles.Source: §C.2 · pp. 16–17

C.1 End-to-End Details

  • Table 4 reports end-to-end metrics averaged over 50 prompts, including fidelity, time, speedup, tile density, refined-tile share, and VBench dimensions.Its caption defines E2E latency as prompt-to-decoded-video time.Source: Table 4 · p. 17

C.3 Qualitative Results on LTX-2.5

The LTX-2.5 comparisons show faster generation with TRIAGE across the displayed resolutions and durations, while frames retain the dense videos’ content, layout, and motion. The examples include a 1440p video with comparatively low PSNR.

C.3 Qualitative Results on LTX-2.5

  • The displayed comparisons span 1088p 30 s, 1440p 15 s, and 4K 7 s videos, with six frames sampled evenly from 10% to 90% of each video.Each video is labeled with generation time, speedup, PSNR against dense output, and its prompt.Source: §C.3 · p. 17, Figure 11 · p. 19
  • 1.77–1.84× faster generation across the displayed LTX-2.5 videos, with content, layout, and motion retained even at the lowest reported PSNR of 26.8 dB.The lowest-PSNR example is the 1440p wizard duel.Source: §C.3 · pp. 17, 20, Figure 11 · p. 19, Figure 12 · p. 20
  • 40.4 dB PSNR and 1.83× speedup are reported for the 4K, 7 s carrier-jet video.The comparison uses dense attention and TRIAGE from the same image, prompt, and seed.Source: Figure 11 · p. 19, §C.3 · p. 21

Related papers

Loading 2610.35706v1…