PaperlayerView exact revision on arXiv

Source-linked AI summary

VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention

Xingyang Li, Dongyun Zou, Shining Zhang, Jiacheng Chen, Haocheng Xi, Lvmin Zhang, Jun-Yan Zhu, Song Han, Zhekai Zhang, Yujun Lin, Muyang Li

arXiv:2609.15810v1cs.CV

TL;DR

Long video-DiT sequences make attention costly, while operand outliers limit low-bit fidelity and FP32 softmax limits throughput. VC-Attention combines clustered, mean-residual value quantization with direct FP8 probability casting. Across four video DiTs, it improves fidelity over low-bit baselines and accelerates attention and end-to-end generation.

  • Problem

    Long video-DiT sequences make attention expensive, while outliers constrain low-bit accuracy and FP32 softmax operations constrain throughput.

  • Method

    VC-Attention combines V-Smooth, which clusters tokens and quantizes mean-subtracted value residuals, with ExpCast-FP8, which directly encodes log-domain scores as E4M3 probabilities.

  • Results

    Across four video DiTs, VC-Attention improves fidelity over low-bit baselines and accelerates attention over BF16 FlashAttention-4 by 1.46–1.59× on datacenter GPUs and 2.3–3.6× on workstation cards.

  • Takeaways & Limitations

    The framework provides a training-free deployment path that targets both value-error and softmax bottlenecks without per-model fitting.

Abstract

from arXiv · show

Diffusion Transformers deliver state-of-the-art video generation, but their long spatiotemporal sequences make attention the dominant deployment cost, and a deployable low-bit kernel must be accurate and fast. Accuracy is limited by outliers: a block's quantization scale is set by its largest entries, leaving typical entries confined to a narrow range of representable values. Prior work smooths queries and keys, but value outliers follow no fixed channel or spatiotemporal structure and remain the dominant source of output error. Speed is limited by softmax: low-bit Tensor Cores accelerate only the two matrix multiplications, so the high-precision exponential between them becomes the longest pipeline stage on datacenter GPUs. We propose VC-Attention, a training-free low-bit attention framework that addresses both by pairing Value smoothing with a fused probability Cast. V-Smooth reorders value tokens by lightweight online clustering, so the tokens in a hardware block quantize well together. It quantizes only the residual after subtracting the block mean, and restores that mean from the row sum the online softmax already maintains. ExpCast-FP8 maps log-domain scores directly to E4M3 probability codes with one fused multiply-add, eliminating the FP32 exponential and the format conversion. We implement VC-Attention for B200, B300, H200, RTX PRO 6000, and RTX 5090. Across Wan2.2, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3, VC-Attention improves fidelity over low-bit baselines, speeds up the attention kernel over BF16 FlashAttention-4 by 1.46-1.59x on datacenter Blackwell and Hopper and by 2.3-3.6x on workstation cards, and generates a clip 1.13-1.19x and 1.36-1.70x faster end to end.

1 Introduction

Video diffusion attention is costly because long token sequences make quadratic attention dominant, while outliers limit low-bit fidelity and softmax limits throughput. VC-Attention addresses both with value smoothing and fused probability casting.

  • 1 Introduction: Attention dominates high-resolution video generation, with a 5-second 720p Wan2.2-14B clip spanning about 70K tokens and exceeding 64% of RTX 5090 generation time.Its quadratic QK and PV arithmetic motivates low-bit Tensor Core acceleration.
  • 1 Introduction: Low-bit accuracy suffers because block scales follow rare outliers, while video attention carries outliers in both operands and value error eventually dominates output error.Prior query-key smoothing leaves value outliers unresolved because they lack fixed channel or spatiotemporal structure.
  • 1 Introduction: VC-Attention combines V-Smooth, which clusters value tokens and quantizes mean-subtracted residuals, with ExpCast-FP8, which directly casts log-domain scores into E4M3 probabilities.V-Smooth restores the block mean from the online softmax row sum without an extra pass or buffer.
  • 1 Introduction: Across four video DiTs, VC-Attention improves fidelity over training-free low-bit baselines while accelerating attention and end-to-end Wan2.2 generation.Attention speedups over BF16 FlashAttention-4 range from 1.46× to 1.59× on datacenter GPUs and from 2.27× to 3.58× on workstation Blackwell.
  • 1 Introduction: 1.60× attention speedup over BF16 FlashAttention-4 and 5.5× over SageAttention2 demonstrates the fused kernel’s deployment advantage.The reported comparison appears in Table 2 and Figure 1.

2 Related Work

Related work accelerates video generation through shorter sampling, reused activations, distributed execution, and shorter latent sequences, while low-bit quantization methods primarily address linear layers and activation outliers. Figure 3 highlights unresolved value-quantization and softmax bottlenecks, which VC-Attention targets without retraining.

  • Efficient video generation: Existing video-generation systems reduce cost through few-step distillation, feature caching, distributed execution, and compact autoencoders that shorten latent sequences.These approaches attack sampling, reuse, distribution, or sequence length rather than the remaining attention stages.
  • Low-bit quantization: Low-bit diffusion quantization has focused mainly on linear layers, using channel rescaling, cross-channel rotation, and related outlier-mitigation strategies.The cited methods include SmoothQuant, AWQ, and QuaRot.
  • Low-bit attention: Figure 3 identifies value quantization as the larger remaining output-error source after QK smoothing, while FP32 softmax exponentiation and casting remain between low-bit matrix products.The Wan2.2 example reports a 1.6× reduction in the probability term from Hadamard rotation without reducing the larger value term.
  • Low-bit attention: VC-Attention addresses video-DiT value error and the scalar probability path on datacenter GPUs without retraining or per-model fitting.Its scope complements prior work that primarily improves the score product.

3 VC-Attention

VC-Attention targets value quantization error and softmax overhead with value-guided smoothing and direct FP8 probability encoding. Its design reorders value tokens, subtracts and restores block means within online softmax, and replaces exponentiation-plus-casting with a fused multiply-add.

  • 3 VC-Attention: VC-Attention combines V-Smooth for value accuracy with ExpCast-FP8 for faster probability casting.V-Smooth reorders keys and values together, while ExpCast-FP8 directly encodes E4M3 probabilities from log-domain scores.
  • 3 VC-Attention: 82% of Wan2.2 output error comes from values after query/key smoothing reduces the first error term.Value outliers vary across tokens, heads, layers, and steps, so rotations and static layouts do not reliably remove them.
  • 3 VC-Attention: The softmax exponential can match or exceed the cost of both FP8 matrix products on H200 and B200, making it a major kernel bottleneck.Tensor Cores execute QK and PV, while exponentials and probability casts use CUDA cores and MUFU, with tile time set by the slower overlapped stage.
  • 3 VC-Attention: V-Smooth preserves attention output by applying the same token permutation to keys and values, then quantizes only block-mean residuals.The mean is restored from the row sum already maintained by online softmax, avoiding an extra pass or buffer; blocks contain 128 rows.
  • 3 VC-Attention: 36% of block energy is removed after sorting, versus 8% in sequence order and 12% under DeltaQuant’s fixed cube.The mean also reduces this head’s E4M3 value error by 1.5×; averaging covers 59.1K blocks across 100 heads.
  • 3 VC-Attention: ExpCast-FP8 uses E4M3’s affine log representation to replace FP32 exponentiation and casting with one fused multiply-add.The direct code matches the conventional byte on 79.6% of a doubling interval and differs by one code on the remainder; per-element relative error reaches 7.5%.

4 Experiments

Across four video DiTs, VC-Attention improves low-bit fidelity while accelerating attention and end-to-end generation across datacenter and workstation GPUs. Ablations show that scheduled grouping and fused preprocessing preserve these gains with limited overhead.

  • 4.2 Main Results: VC-Attention remains closer to BF16 attention than training-free baselines across four video DiTs at both 8-bit and 4-bit precision.At 8 bits it leads every fidelity column, while at 4 bits it improves PSNR over SageAttention3 by 2.9 dB on Wan2.2 and 3.6 dB on LongCat-Video.
  • 4.2 Main Results: V-Smooth improves matched-precision PSNR by 1.1–2.8 dB at 8 bits and 0.5–3.6 dB at 4 bits, whereas QK Hadamard changes PSNR by at most 0.1–0.3 dB.These results identify value quantization rather than QK quantization as the remaining dominant fidelity issue.
  • 4.3 Efficiency: 1.59× and 1.46× attention speedups over BF16 FlashAttention-4 occur on B200 and H200, while 2.27× and 3.58× occur on RTX PRO 6000 and RTX 5090.End-to-end Wan2.2 clip speedups are 1.19×, 1.13×, 1.36×, and 1.70× on those cards, respectively.
  • 4.3 Efficiency: V-Smooth adds only 3–4% of attention time, while the B300 implementation reaches 1.47× BF16 speedup and 18.4 dB PSNR versus 1.31× and 17.1 dB for a naive FP8 kernel.The B300 comparison uses FP8 for both QK and PV because the card lacks INT8 QK matrix multiplication.
  • 4.4 Ablation Study: Grouping the first 25% of denoising steps costs only 0.5 dB versus grouping every step, while uniform-quarter grouping falls 2.6 dB behind.The leading-quarter schedule saves 16 s per clip, reducing latency from 367.0 s to 351.0 s.
  • 4.4 Ablation Study: Fusing preprocessing stages raises the cumulative speedup to 8.74×, leaving grouping at 29% of the 4.8 ms preprocessing-chain cost per attention call.Quantizer fusion, smoothing and permutation, rotary embedding, and handwritten kernels contribute successively to the reduction.

5 Conclusion

VC-Attention targets the two non-matrix-multiplication bottlenecks in low-bit video-DiT attention: value quantization and softmax. Its value smoothing and fused probability casting improve fidelity over training-free baselines while delivering substantial attention speedups across evaluated GPUs.

  • 5 Conclusion: VC-Attention combines online token grouping and mean-residual value quantization with fused FP8 probability casting for training-free low-bit video attention.V-Smooth restores the block mean through the online softmax recurrence, while ExpCast-FP8 replaces exponential computation and casting with one fused multiply-add.
  • 5 Conclusion: Across Wan2.2, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3, VC-Attention stays closer to full-precision attention than same-bit training-free baselines.Attention speedups over BF16 FlashAttention-4 reach 1.59× on B200, 1.46× on H200, 2.27× on RTX PRO 6000, and 3.58× on RTX 5090.

A Evaluation Protocol Details

The evaluation fixes generation settings, baselines, timing procedures, and hardware-specific scope for the reported attention and end-to-end measurements.

  • Evaluation Protocol Details: The four video settings contain 75.6K, 37.4K, 111.7K, and 73.5K attention tokens for Wan2.2, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3, respectively.Methods share prompts and random seeds within each setting.
  • Evaluation Protocol Details: Attention speedup uses FlashAttention-4 latency divided by evaluated-method latency, while end-to-end speedup includes prompt processing, denoising, and VAE decoding.Kernel timing excludes quantization, token grouping, and preprocessing; end-to-end timing includes their cost but excludes model loading.
  • Evaluation Protocol Details: FlashAttention-4 is the BF16 fidelity and speed reference on every reported card, whereas SageAttention2's B200 kernel is recompiled for sm 100a.Official baseline configurations are used where available.
  • Evaluation Protocol Details: ExpCast-FP8 is restricted to the 8-bit kernel on B200 and H200 because its E4M3 mapping does not apply to NVFP4's E2M1 probability format.These platforms are the settings where low-bit matrix multiplication outpaces the scalar softmax pipeline.
  • Evaluation Protocol Details: V-Smooth grouping and demeaning run during the first quarter of denoising steps, recomputing permutations every four adjacent steps.The schedule is keyed to the step index and uses the same fractional rule across models.

B Fused Preprocessing, Operand by Operand

The fused preprocessing chain combines operand-specific transformations with quantization, avoiding high-precision intermediate writes to HBM.

  • Fused Preprocessing, Operand by Operand: The three preprocessing stages fuse quantization and padding with QK Hadamard rotation, smoothing means with gather, and rotary embedding.The second stage combines key-channel and per-block value means; the final stage writes no high-precision intermediate to HBM.
  • Fused Preprocessing, Operand by Operand: K fuses centering statistics, RoPE, gathering, Hadamard rotation, mean subtraction, and quantization before writing its result.V separately fuses gathering, block demeaning, and quantization while storing block means with residual codes.
  • Fused Preprocessing, Operand by Operand: At 8 bits, value means are stored divided by the per-channel scale so one epilogue multiply handles both terms of Equation (6).At 4 bits, NVFP4 microscales are applied inside the matrix instruction and the means are added unscaled.

C Where the ExpCast-FP8 Constant Comes From

The ExpCast-FP8 offset β is chosen as a minimax correction to the straight-line log approximation error over one doubling.

  • Where the ExpCast-FP8 Constant Comes From: The signed E4M3 coding error spans [−0.6886, 0], so halving the range yields the minimax shift β = −0.35.The resulting shift centers the error within a ±0.34-code envelope.

D Proof of Proposition 3.1

The proof bounds probability distortion from ExpCast-FP8 by analyzing per-entry decoding ratios, endpoint reweighting, and low-score tail mass before transferring the result to attention outputs.

  • Proof of Proposition 3.1: The proof begins from an exact softmax row expressed through wk = 2^u_k and p_k = w_k divided by the row sum.E4M3 decoding after removing the common 2^8 scale supplies the approximate weights.
  • Proof of Proposition 3.1: For normal entries, the exact mantissa coordinate is 8(2^f_k − 1), whereas the direct map uses 8f_k, producing the analyzed approximation gap.The proof writes z_k = u_k + 8 = n_k + f_k with f_k in [0, 1).
  • Proof of Proposition 3.1: ExpCast-FP8's log-domain approximation error is bounded by the Mitchell straight-line error scaled across the eight E4M3 codes per doubling.Its minimax correction uses β = −0.35 after halving the [0, 0.6886] error range.
  • Proof of Proposition 3.1: The decoded-to-exact ratio is checked between rounding cells by evaluating both one-sided limits at rounding boundaries, including the carry cell.This establishes the ratio behavior used before the row-level variation bound.
  • Proof of Proposition 3.1: The maximum total variation over the decoded-to-exact ratio interval is attained by assigning endpoint ratios to different portions of the probability mass.The interval is defined by a = 0.9290 and b = 1.0746.
  • Proof of Proposition 3.1: The probability-path bound is TV(p, bp) < 0.0364 + τ, where τ captures mass in entries with uk < −14.The proof separately notes that residual-value quantization contributes an additional error term.

E Qualitative Results

VC-Attention is compared with BF16 FlashAttention-4 and SageAttention baselines across eight qualitative video-generation configurations. The evaluation controls prompt, seed, and frame indices, while reporting clip-level PSNR against each clip’s FlashAttention-4 render.

  • Qualitative protocol: Controlled strips compare FlashAttention-4, SageAttention2, and VC-Attention using the same prompt, seed, and source frame indices.This controls sampling differences so visual changes down a column reflect the attention method.
  • Evaluation metric: PSNR is computed for each clip against its own FlashAttention-4 render over every frame.This makes the metric a clip-level fidelity comparison relative to the BF16 reference for that clip.
Loading 2609.15810v1…