PaperlayerView exact revision on arXiv

Source-linked AI summary

QuantWM: Temporally Consistent 2-Bit KV Cache Quantization for World Models and Video Generation

Jiaqi Zhao, Xiaobin Hu, Bo Yin, Junpeng Jiang, Miao Zhang, Shuicheng Yan

arXiv:2609.26425v2cs.CVcs.AI

TL;DR

Video-generation and world-model KV caches create a substantial memory burden, while 2-bit quantization can degrade visual quality despite nearly unchanged benchmark scores. QuantWM uses training-free clustering and attention compensation to reduce attention shifts during 2-bit cache quantization. Across evaluated models, it improves visual quality and temporal consistency, with up to 6.20× KV cache memory compression.

  • Problem

    KV caches are memory-intensive for video generation and world models, while existing 2-bit methods can show nearly lossless video-benchmark performance despite visible visual degradation.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Method

    QuantWM is a training-free 2-bit KV cache quantization framework that combines Query-sensitive centroid selection with low-rank attention compensation.

    Source: §1 · p. 2, §4 · p. 4, §4.1 · p. 6

  • Results

    Across video-generation and world models, QuantWM improves visual quality and temporal consistency, with up to 6.20× KV cache memory compression.

    Source: §1 · p. 3, §5.2 · pp. 7–8, §6 · p. 10

  • Takeaways & Limitations

    The reported results show that QuantWM can reduce KV cache memory while maintaining strong VBench performance and improving visual quality and temporal consistency.

    Source: §1 · p. 3, §5.2 · p. 8

Abstract

from arXiv · show

KV cache memory has become a major deployment bottleneck for video generation and world models, which motivates low-bit quantization study for efficiency. Existing 2-bit KV cache quantization methods can achieve nearly lossless performance on video benchmarks such as VBench, however, we find that they still cause severe temporal flickering and visual degradation. Meanwhile, deeper investigates show that Key quantization produces smaller reconstruction errors than Value, but surprisingly leads to much larger output degradation. We trace this discrepancy to attention: small Key perturbations can change the attention logits, i.e., QK^\top, and shift the temporal-spatial tokens selected by Queries. These observations motivate us to explicitly preserve attention logits and temporal-spatial token selection during KV cache quantization to alleviate the visual degradation problem. To address this issue, we present QuantWM, a training-free and strictly causal 2-bit KV cache quantization framework. QuantWM introduces two complementary techniques to mitigate the attention shifts. Firstly, quantization-sensitivity-aware clustering (QSAC) jointly considers historical Query sensitivity and residual ranges to select INT2-friendly Key centroids, which reduces quantization errors in channels that are more critical to attention. In addition, principal-subspace attention compensation (PSAC) restores the remaining Key errors along the dominant Query subspace using low-rank projections, which provides a direct and efficient correction to stabilize attention logits. Extensive experiments on Causal-Forcing, LingBot-World-v2, HY-World 1.5, Matrix-Game-2 and Longcat-Video demonstrate that QuantWM significantly improves visual quality and temporal consistency, while outperforming existing methods across image and video quality metrics with up to 6.20x KV cache memory compression and limited additional overhead.

1 INTRODUCTION

QuantWM targets the memory burden of video-generation KV caches and the visual degradation that 2-bit methods can hide behind nearly unchanged benchmark scores. It preserves attention behavior and improves visual quality and temporal consistency while compressing the cache.

  • Existing 2-bit quantization can score nearly losslessly on video benchmarks while still producing flickering, blurring, and visual artifacts.LingBot-World-v2 requires more than 21 GiB of KV cache memory to generate a 5-second video of about 93 frames.Source: §1 · pp. 1–2
  • Small Key perturbations can alter QK⊤ attention logits, shifting which temporal-spatial tokens Queries select.Those shifts can redirect Queries to incorrect cached frames or spatial regions, leading to flickering and visual degradation.Source: §1 · p. 2
  • QuantWM is training-free and combines QSAC centroid selection using Query sensitivity and residual ranges with PSAC low-rank correction along dominant Query directions.Both techniques are designed to reduce Key quantization's effect on attention logits.Source: §1 · p. 2
  • Up to 6.20× KV cache memory compression accompanies improved temporal consistency and visual quality across video-generation and world models.The results also report strong VBench performance with limited inference overhead.Source: §1 · p. 3

2 PRELIMINARY

The preliminary section introduces affine quantization parameters and the centroid-residual representation used for KV cache quantization.

  • Affine b-bit quantization and dequantization are formulated for a tensor X.The supplied passage introduces the formulation but does not include its equation.Source: §2 · p. 3
  • The affine quantization parameters s and z denote the scaling factor and zero-point, respectively.Quantization parameters may be shared per token, per channel, or per group.Source: §2 · p. 3
  • In the QVG centroid-residual representation, each Key token is assigned to a K-means centroid and its residual is the token minus that centroid.The centroid is preserved at BF16 while the residual is quantized to low-bit.Source: §2 · p. 3

3 MOTIVATION

The motivation section identifies a mismatch between benchmark scores and visible video degradation, traces the degradation to Key quantization, and links Key perturbations to attention-driven token-selection shifts.

3.1 THE OVERLOOKED VISUAL DEGRADATION UNDER 2-BIT KV QUANTIZATION

  • Existing 2-bit KV methods can retain nearly lossless benchmark scores while producing flickering, blurring, and visual artifacts relative to BF16.The section identifies this mismatch as a reason to investigate which cache component drives the degradation.Source: §3.1 · p. 3

3.2 KEY QUANTIZATION DOMINATES VISUAL DEGRADATION

  • K2-only causes more visual degradation than V2-only, and K2V2 performs similarly to K2V16.The reported frame-level PSNR, SSIM, and LPIPS comparisons show K16V2 clearly ahead of K2V16.Source: §3.2 · p. 4, Table 2 · p. 4
  • Key reconstruction error is smaller than Value reconstruction error, even though Key quantization produces larger output error.This indicates that Key sensitivity is not explained by quantization-error magnitude alone.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 KEY QUANTIZATION PERTURBS TEMPORAL-SPATIAL TOKEN SELECTION

  • Small Key perturbations may reorder attention logits and shift the temporal-spatial tokens selected by Queries.The visualized shifts correspond to Queries attending to different historical frames or spatial regions, explaining how smaller Key errors can still accompany greater output degradation.Source: §3.3 · p. 4

4 QUANTWM

QuantWM is a training-free 2-bit KV cache quantization framework that reduces attention shifts with QSAC and PSAC. QSAC selects centroids using Query sensitivity and residual quantization difficulty, while PSAC corrects remaining logit perturbations along dominant Query directions.

4 QUANTWM

  • QuantWM combines QSAC to reduce Key-quantization effects on attention logits with PSAC to correct remaining attention shifts.The framework is training-free and targets 2-bit KV cache quantization for video generation and world models.Source: §4 · p. 4

4.1 QUANTIZATION-SENSITIVITY-AWARE CLUSTERING (QSAC)

  • QSAC selects Key centroids using historical Query sensitivity and residual quantization difficulty to reduce expected perturbations to QK^T.It uses historical Queries to approximate channel-wise sensitivity and considers residual-group dynamic ranges when selecting among candidate centroids.Source: §4.1 · pp. 5–6

4.2 PRINCIPAL-SUBSPACE ATTENTION COMPENSATION (PSAC)

  • PSAC restores remaining Key quantization errors along dominant Query directions to correct attention-logit shifts.The top eight Query directions contain more than 60% of total Query energy, and the implementation uses rank r = 8 with limited additional overhead.Source: §4.2 · pp. 6–7

5 EXPERIMENTS

Across five video-generation and world models, QuantWM improves frame-level quality, reduces temporal-spatial token-selection shifts, and compresses KV cache memory with limited latency impact. Experiments also compare VBench performance, visualizations, and an HY-World 1.5 ablation.

5.1 EXPERIMENTAL SETUP

  • Experiments cover five open-source models at 480p and 720p with 93-frame videos, using PSNR, SSIM, LPIPS, VBench, and QVG and KIVI baselines.The evaluation uses a prompt suite from MovieGen Benchmark, with official first-frame images and action sequences for Matrix-Game-2.Source: §5.1 · p. 7
  • QuantWM uses streaming chunkwise KV cache quantization, quantizing each cache chunk once after construction; PSAC stores its top-8 Query-subspace eigenvectors in BF16 and quantizes C to INT8.Source: §5.1 · p. 7

5.2 MAIN RESULTS AND VISUALIZATIONS

  • QuantWM outperforms QVG and KIVI on all frame-level metrics across five models; on LongCat-Video, PSNR rises from 20.683 to 25.508 and LPIPS falls from 0.0784 to 0.0332.Table 3 covers visual quality and VBench results on 480p videos.Source: §5.2 · p. 7, Table 3 · p. 8
  • Across all five models, QuantWM's VBench scores exceed QVG and remain comparable to BF16, while visualizations show clearer details and less flickering than QVG.The paper notes that video benchmarks alone may not fully reflect quantization-related visual degradation.Source: §5.2 · p. 8, Figure 5 · p. 8

5.3 TEMPORAL-SPATIAL TOKEN SELECTION SHIFTS RATIO

  • For the top-1 temporal-spatial token-selection shift ratio, QVG reaches 61.36%, while QuantWM reduces shifts by up to 10.19% across the five models.The comparison is against BF16 and evaluates shifts in selected tokens.Source: §5.3 · p. 8, Table 4 · p. 9

5.4 ABLATION STUDY

  • Table 5 reports an ablation study of QuantWM on HY-World 1.5.Source: Table 5 · p. 9

5.5 INFERENCE EFFICIENCY

  • For 93-frame generation, KV cache compression reaches 6.20×; LongCat-Video memory falls from 21.709 GiB to 3.625 GiB.Figure 6 compares BF16 and QuantWM memory across models.Source: §5.5 · p. 9, Figure 6 · p. 9
  • Generation latency rises 5.44% on LingBot-World-v2 and 5.87% on HY-World 1.5, but falls 17.78% on LongCat-Video.The paper characterizes the overall latency impact as limited; Table 6 compares BF16 and QuantWM.Source: §5.5 · p. 9, Table 6 · p. 9

6 CONCLUSION

The conclusion links Key quantization errors to attention-logit and token-selection shifts, and presents QuantWM as a training-free 2-bit approach to mitigate them. The supplied background passages situate the work in video-world-model memory needs and prior quantization methods.

6 CONCLUSION

  • Key quantization is more sensitive than Value because Key perturbations can alter attention logits and shift temporal-spatial token selection.Source: §6 · pp. 9–10
  • QuantWM combines QSAC, which uses Query sensitivity and residual quantization difficulty for centroid selection, with PSAC, which corrects remaining logit perturbations along dominant Query directions.The framework is training-free and uses 2-bit KV cache quantization.Source: §6 · p. 10
  • Experiments across video-generation and world models show that QuantWM alleviates flickering and visual degradation while maintaining high KV cache compression.Source: §6 · p. 10

A.1 VIDEO GENERATION AND WORLD MODELS

  • Causal video models use KV caches for historical representations, and their caches grow with context, making cache efficiency a central deployment bottleneck.Source: §A.1 · p. 13

A.2 MODEL QUANTIZATION

  • The related-work discussion describes post-training quantization methods for compressing model weights and activations, including GPTQ, SmoothQuant, and AWQ.Source: §A.2 · p. 13
  • Prior KV cache quantization methods target language models, while QVG extends 2-bit quantization to autoregressive video diffusion but may leave visual artifacts unreflected by benchmark scores.The paper distinguishes its focus on attention-logit perturbations and temporal-spatial token selection from prior work correcting softmax attention bias.Source: §A.2 · p. 13

B.1 RESULTS ON 720P 93-FRAMES VIDEOS

At 720p, QuantWM's frame-level quality improvements over QVG persist across the evaluated models, while its VBench performance remains comparable to BF16.

B.1 RESULTS ON 720P 93-FRAMES VIDEOS

  • At 720p, QuantWM improves frame-level visual quality over QVG across evaluations and has the best overall frame-level quality across evaluated models; on Matrix-Game-2, PSNR rises from 17.721 to 20.568 and LPIPS falls from 0.3047 to 0.1651.VBench performance remains comparable to BF16; Table 7 reports 720p visual-quality and VBench comparisons.Source: §B.1 · p. 13, Table 7 · p. 14

B.2 RESULTS ON LONG-VIDEO GENERATION

The supplied long-video and appendix results cover one-minute VBench performance, qualitative comparisons, Query-energy distributions, PSAC rank, and causal initialization of historical statistics.

B.2 RESULTS ON LONG-VIDEO GENERATION

  • Across five models, QuantWM improves average VBench scores over QVG in one-minute generation; on HY-World 1.5, the score rises from 0.5949 to 0.6627.Table 8 reports VBench results for one-minute video generation.Source: §B.2 · pp. 14–15

B.3 MORE VISUALIZATIONS OF GENERATED VIDEOS

  • Across models and scenarios, visualizations show QVG flickering, blurring, and local artifacts, while QuantWM better preserves object appearance, spatial detail, and temporal consistency.Figure 7 provides additional visual comparisons.Source: Figure 7 · p. 15, §B.3 · p. 16

C.2 THE EFFECT OF THE HYPER-PARAMETER r IN PSAC

  • Increasing PSAC rank generally improves frame-level quality with diminishing gains; moving from rank 8 to 16 adds 0.1341 bits on HY-World 1.5 and 0.2569 bits on Causal-Forcing, motivating rank 8 as a trade-off.Table 9 summarizes the rank ablation and storage overhead.Source: §C.2 · p. 16, Table 9 · p. 16

C.3 INITIALIZATION OF HISTORICAL QUERY STATISTICS

  • During the first cache chunk, QSAC uses uniform channel sensitivity and PSAC remains inactive; later chunks update historical statistics only from previously generated content.The approach preserves strict causality and requires no calibration stage or manually designed warm-up schedule; Table 10 studies QSAC's diagonal approximation.Source: §C.3 · pp. 16–17, Table 10 · p. 17

C.1 QUERY ENERGY DISTRIBUTION ON MORE MODELS

  • Figure 8 plots cumulative Query energy across models and layers.Source: Figure 8 · p. 17

C.4 THE EFFECT OF THE DIAGONAL QUERY APPROXIMATION IN QSAC

The QSAC ablations find no consistent frame-quality gain from retaining the full Query matrix, while increasing candidate-centroid evaluation beyond M = 4 adds latency without quality improvement.

C.4 THE EFFECT OF THE DIAGONAL QUERY APPROXIMATION IN QSAC

  • QSAC uses the historical Query second-moment matrix’s diagonal entries for its Query-aware distance; the full-form ablation substitutes a quadratic form to select top-m candidate centroids.The remaining QuantWM pipeline is unchanged in that ablation.Source: §C.4 · pp. 17–18
  • The full Query matrix provides no consistent frame-quality improvement and adds inference latency, so QSAC uses the diagonal approximation.Differences in PSNR, SSIM, and LPIPS are marginal on HY-World 1.5 and Causal-Forcing.Source: §C.4 · p. 18

C.5 THE EFFECT OF THE NUMBER OF CANDIDATE CENTROIDS IN QSAC

  • Evaluating all 256 centroids does not improve PSNR, SSIM, or LPIPS over M = 4 on either model, while significantly increasing inference latency.Increasing M beyond 4 brings only marginal changes in frame-level quality; M = 4 is used as the quality-efficiency trade-off.Source: §C.5 · p. 18, Table 11 · p. 18

D.1 IMPLEMENTATION DETAILS

QuantWM compresses completed cache chunks and reconstructs BF16 keys and values on demand for native attention. Its latency effect depends on cache access: reduced data movement can outweigh quantization overhead for offloaded caches, while GPU-resident caches may see modest overhead.

D.1 IMPLEMENTATION DETAILS

  • At each cache-chunk boundary, QuantWM packs completed BF16 chunks and releases their original storage, while the active chunk remains BF16.Per-head Query statistics are maintained incrementally, and the principal basis is computed at chunk boundaries without retaining historical Query tensors.Source: §D.1 · p. 18
  • Packed storage uses four INT2 residual codes per byte, uint8 centroid indices, and bit-packed INT2 zero-points, with scales and PSAC coefficients stored using quantized formats.For each chunk, QuantWM selects the smaller dense or sparse coefficient layout, including sparse-index storage.Source: §D.1 · p. 19
  • A fused Triton kernel reconstructs BF16 keys and values on demand, combining residual unpacking, parameter reconstruction, centroid gathering, and PSAC accumulation.Native FlashAttention or SDPA consumes temporary reconstructed tensors that are not retained in the persistent cache.Source: §D.1 · p. 19, Figure 9 · p. 19

D.2 THE IMPACT OF KV CACHE QUANTIZATION ON LATENCY IS MODEL-DEPENDENT

  • For LongCat-Video, reducing repeated host–GPU KV-cache transfers lowers end-to-end latency, whereas GPU-resident caches may incur modest overhead.The latency impact depends on cache access and movement as well as compression ratio.Source: §D.2 · p. 19
Loading 2609.26425v2…