Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches

1 INTRODUCTION

• Combining query-channel sparsity with low-precision keys reduces both the number of channels read and the bits per key element.The paper identifies reducing storage and repeated cache access as requirements for efficient long-context inference.Source: §1 · p. 1
• Key quantization favors flatter channel scales, whereas query-channel pruning favors sharp query energy; a shared rotation can compromise the latter while addressing key outliers.Q-PCA concentrates query energy but does not explicitly control key outliers; Hadamard rotation mitigates key outliers but can disperse query energy.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Dual-QK pairs an invertible query transform with its inverse transpose on keys, preserving query–key dot products before quantization and pruning while shaping their distributions differently.This paired non-orthogonal design targets balanced key scales and concentrated query energy.Source: §1 · p. 2
• Calibrated query and key statistics guide partial key whitening and a query-aligned basis to balance key scales for INT2 quantization and concentrate query energy.The resulting representation supports dynamic channel selection based on current query magnitudes.Source: §1 · p. 2
• Dual-QK retains channel-0 in BF16 and excludes it from range estimation, while bucket-relative RoPE keeps stored-key positional offsets within the calibration window.The corresponding query adjustment preserves attention scores before quantization and pruning.Source: §1 · p. 2
• At 40% query-channel sparsity, Dual-QK improves accuracy over OSCAR on most tasks; at 128K context, it provides 6.8× KV-cache compression and an estimated 8.3× reduction in KV read volume relative to unpruned BF16.Evaluations cover four models, generative accuracy, long-context retrieval, and decoding throughput.Source: §1 · p. 2

2.1 KV CACHE QUANTIZATION

• KV-cache quantization methods address range and outlier effects through per-channel or per-token quantization, mixed precision, fine-grained grouping, rotations, channel reordering, clipping, or random projections.Examples include KIVI, KVQuant, rotation-based methods, TurboQuant, OSCAR, and serving-oriented approaches.Source: §2.1 · pp. 2–3

2.2 SPARSE ATTENTION

• Sparse-attention methods select tokens or pages, or select channels or low-dimensional key representations; Dual-QK jointly designs Q/K coordinates for low-bit key quantization and dynamic channel selection.Token-level approaches include eviction and query-aware access; channel-level approaches include ThinK, SparQ Attention, Double Sparsity, Loki, and low-rank projections.Source: §2.2 · p. 3

3.1 DUAL TRANSFORMS

• Calibrated partial whitening and a query-principal basis balance key scales and concentrate query energy while preserving query–key scores before compression.The whitening strength α controls the trade-off; the transforms are fixed per layer and KV head after calibration.Source: §3.1 · pp. 3–4

3.2 DYNAMIC QUERY-CHANNEL PRUNING

• For each query, Dual-QK retains the m channels with largest group-level query norms under one shared mask, while all key channels remain available to later queries.Query magnitude is a useful proxy when key channels have comparable scales, but it is not an exact measure of attention-score error.Source: §3.2 · p. 4

3.3 INT2 KV QUANTIZATION

• Channel 0 stays BF16 and outside the range estimate; remaining keys use asymmetric INT2, with calibrated reciprocal rescaling preserving fixed-channel-set scores before quantization.Values follow OSCAR’s calibrated rotation, clipping, and INT2 recipe; the first 64 and most recent 256 tokens remain BF16. INT2 keys use channel-major storage so decoding can load selected rows.Source: §3.3 · p. 5

3.4 BUCKET-RELATIVE ROPE

• Bucket-relative RoPE expresses stored keys relative to local anchors, keeping their positional offsets within the calibration window while reusing the same dual transform pair.With bucket width w = 2048, the query’s relative position can still exceed the window; masks may differ across buckets, whose scores share one global softmax.Source: §3.4 · p. 5

4.1 EXPERIMENTAL SETTINGS

• Evaluations cover four models, five generative benchmarks, and RULER retrieval from 4K to 128K; quantized methods share 40% pruning and retain the first 64 and most recent 256 KV entries in BF16.Dual-QK is calibrated on WikiText-2 and evaluated with transforms reused across benchmarks and context lengths.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• At 128K, Dual-QK retains 57.0–64.4% accuracy across Llama and the two Qwen models, versus OSCAR’s 2.0–25.2%.The comparison uses 40% query-channel sparsity and comparable KV storage and read budgets.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 ABLATION STUDIES

• As pruning rises, Dual-QK’s accuracy gap over OSCAR widens; it remains ahead on both tasks and models at 50%, but HumanEval accuracy drops sharply at 60%.Figure 3 covers GPQA-Diamond and HumanEval on Qwen3-8B and Qwen3-4B-Thinking-2507, with unpruned BF16 references and three-seed error bars.Source: Figure 3 · p. 7, §4.3 · pp. 7–8

4.2 MAIN RESULTS

• At 40% sparsity, Dual-QK improves generative accuracy over OSCAR on most tasks; without pruning, its mean differs from OSCAR by under one point on Llama-3.1-8B-Instruct and Qwen3-8B.At 40% sparsity, Dual-QK uses 2.34 capacity BPE versus OSCAR’s 2.28.Source: §4.2 · p. 7, §4.3 · p. 8, Table 4 · p. 8

4.3 ABLATION STUDIES

• On Qwen3-8B at 40% sparsity, removing channel-0 protection or bucket-relative RoPE lowers HumanEval accuracy from 90.6% to 71.5% and 51.4%, respectively.Q-PCA alone lowers LiveCodeBench from 36.2% to 15.3% and AIME25 from 53.3% to 17.8%.Source: Table 3 · p. 8, §4.3 · p. 8
• At 40% sparsity, calibration-dataset spreads are 0.5 points on MATH-500, 2.0 on HumanEval, 2.1 on LiveCodeBench, and 5.6 on AIME25.WikiText-2 calibration is competitive on most tasks, while AIME25 is more sensitive to the calibration source.Source: §4.3 · p. 8, Table 5 · p. 9

4.4 THROUGHPUT

• With batch size 1, Dual-QK reaches 90 tokens/s at 90K on Qwen3-4B-Thinking-2507, versus BF16’s 24 tokens/s, a 3.75× speedup.The measurements include transformations and cache operations during decoding.Source: §4.4 · p. 8, Figure 4 · p. 9
• At 90K, quantized methods support larger tested batches than BF16 on Llama and both Qwen models; Dual-QK maintains higher aggregate throughput than OSCAR at feasible batch sizes.On Ministral, both quantized methods support batch size 8, while all methods run out of memory at batch size 16.Source: Figure 4 · p. 9, §4.4 · p. 9

• At small batch sizes, selection overhead can offset savings from skipped key reads; pruning benefits grow when KV-cache access occupies more of decoding, such as at larger batches and longer contexts.Long-context accuracy gaps to BF16 remain, and query offsets relative to older buckets can exceed the 2K calibration window.Source: §5 · p. 9
• Bucket-relative RoPE bounds stored-key offsets but does not guarantee matching activation statistics; calibration over broader positional ranges is left for future work.Source: §5 · p. 9

6 CONCLUSION

• Dual-QK pairs query and key transforms to balance key scales and concentrate query energy while preserving attention scores before compression.The conclusion frames joint coordinate design as valuable for quantization and channel selection.Source: §6 · p. 9

A ADDITIONAL RELATED WORK

• Unlike CommVQ and NOVA-KV’s vector-quantization approaches, Dual-QK designs coordinates for scalar INT2 quantization and dynamic channel selection.Both NOVA-KV and Dual-QK use query statistics to guide key compression.Source: A ADDITIONAL RELATED WORK · p. 14
• Unlike GEAR and STAR-KV, Dual-QK retains the full transformed channel dimension; its query-specific mask controls which key coordinates are read.Unselected coordinates remain available at later steps.Source: A ADDITIONAL RELATED WORK · p. 14
• LeanK and SparK reduce stored key representations, whereas Dual-QK stores every transformed key channel at low precision and computes a shared read mask from current GQA queries.Source: A ADDITIONAL RELATED WORK · p. 14
• KVzap prunes token positions, while Dual-QK retains all positions and reduces key precision and the channels read; combining them would require deciding which positions to retain.Source: A ADDITIONAL RELATED WORK · p. 14

B.1 MODELS AND EVALUATION

• Evaluation covers eight RULER NIAH subtasks with 25 examples each at contexts from 4K to 128K, using greedy decoding capped at 128 tokens and equal subtask averages; LongBench uses greedy decoding and one run per configuration.Main generative results report mean ± standard deviation over five runs for Llama and Qwen3 and three for Ministral-3. RULER scores average three independently generated datasets.Source: §B.1 · p. 14

B.2 CALIBRATION

• Calibration statistics come from 128 non-overlapping 2,048-token WikiText-2 windows, with post-RoPE queries and keys accumulated in FP64 per layer and KV head.Transforms are computed in FP32 with ϵ = 10−5 tr(CK)/d.Source: §B.2 · pp. 14–15
• Diagonal refinement fits per-layer, per-head scales to reduce query-weighted key error, retaining the refinement only when validation error improves over D = I.It uses sampled fitting and validation keys, then folds the final scales into A and B.Source: §B.2 · p. 15
• Whitening strength α is selected from 0 to 1 using held-out 8K and 32K inputs to minimize relative squared decoder-hidden-state error, averaging 0% and 40% sparsity.The selected transforms are reused across benchmarks and context lengths.Source: §B.2 · p. 15
• Values use OSCAR’s calibrated rotation composed with a Hadamard transform and per-token asymmetric INT2 with a 0.92 clipping ratio; Dual-QK keys are not clipped.The calibration-domain ablation refits components using WikiText-2, MMLU, or GPQA-Diamond.Source: §B.2 · p. 15

C.1 DUAL TRANSFORMS

• Partial whitening reduces the spread of key energies, while the query-aligned basis orders compensated query energy; the identities use uncentered moments and do not require zero-mean queries.The transformed key-moment eigenvalue is µ_i(µ_i + ϵ)^−α, approaching µ_i^(1−α) when µ_i ≫ ϵ.Source: §C.1 · p. 15
• Stronger whitening makes key energies more uniform but also changes query scaling, so α is selected using hidden-state error to account for quantization and pruning together.Figure 5 presents channel profiles across models; Table 8 reports relative hidden-state error for selecting α, averaged over 0% and 40% sparsity.Source: §C.1 · p. 15, Figure 5 · p. 16, Table 8 · p. 16

C.2 CHANNEL SELECTION

• Group-L2 retains the most query energy for a fixed channel budget, but partial whitening does not guarantee the centered-key covariance condition, making query energy a pruning-error proxy.The criterion sums query energy over query heads in a GQA group; selection occurs before diagonal refinement, so scaling a query channel and inversely scaling its key channel leaves the product unchanged.Source: §C.2 · p. 16
• Table 9 reports generative accuracy without query-channel pruning, while Table 7 lists the evaluation datasets.The supplied table captions specify the scope but do not provide the underlying scores.Source: Table 7 · p. 15, Table 9 · p. 17

C.3 KEY QUANTIZATION ERROR

• The calibration objective assumes key quantization error is independent of the query, and squared attention-logit error includes an additional factor of 1/d.The expression motivates the calibration objective but does not model error propagation during autoregressive decoding.Source: §C.3 · p. 17
• A channel with modest reconstruction error can still contribute substantially to score error when its query energy is high; channel-0 protection and diagonal refinement target this query-weighted error.Channel-0 is retained in BF16 and excluded from range estimation, narrowing the INT2 range when it sets a row extremum.Source: §C.3 · p. 17

C.4 BUCKET-RELATIVE ROPE

• With w = 2048, bucket-relative RoPE keeps stored-key offsets within the calibration window, but this does not guarantee matching activation statistics and older-bucket query offsets may exceed the window.Anchor adjustments use each checkpoint’s rotary frequencies, including YaRN for Qwen3-8B beyond 32K and for Ministral.Source: §C.4 · p. 17

D.1 RESULTS WITHOUT CHANNEL PRUNING

• At 0% query-channel sparsity, Dual-QK maintains competitive accuracy without pruning, with five-benchmark averages within one percentage point of OSCAR on Llama-3.1-8B-Instruct and Qwen3-8B.Source: §D.1 · p. 17
• Without pruning, Dual-QK has the highest mean among quantized methods on Llama-3.1-8B-Instruct and Qwen3-8B.Source: §D.1 · p. 17
• On Qwen3-4B-Thinking-2507, Dual-QK remains close to BF16 on GPQA-Diamond and HumanEval, while larger gaps remain on LiveCodeBench and AIME25.Source: §D.1 · p. 17

D.2 LONGBENCH RESULTS

• LongBench evaluation uses eight full test sets with greedy decoding and task-specific generation limits; inputs longer than 31.5K tokens are truncated in the middle.Qasper, QMSum, and MultiNews use chat templates, and thinking is disabled for Qwen3-8B.Source: §D.2 · p. 18
• At 40% sparsity, Dual-QK’s LongBench means exceed OSCAR by 10.08 points on Llama-3.1-8B-Instruct and 3.12 points on Qwen3-8B.Increasing sparsity from 0% to 40% reduces Dual-QK’s mean by 1.54 and 1.22 points, compared with 11.32 and 4.09 for OSCAR, respectively.Source: §D.2 · p. 18

D.3 COMPARISON WITH THINK

• On LongBench, Dual-QK exceeds OSCAR + ThinK by 5.56 and 6.58 mean-score points on the two models, and is within 0.53 points of KIVI-2 + ThinK.At 128K context, Dual-QK’s bandwidth cost is 1.93 versus 2.41 bits per KV element for KIVI-2 + ThinK.Source: §D.3 · p. 18, Table 12 · p. 19
• On generative benchmarks, Dual-QK has higher mean GPQA-Diamond and HumanEval accuracy than both ThinK combinations on both models.Table 13 reports accuracy percentages as mean ± standard deviation over three seeds.Source: §D.3 · p. 18, Table 13 · p. 19