PaperlayerView exact revision on arXiv

Source-linked AI summary

RoPE-Aware Bit Allocation for KV-Cache Quantization

Fengfeng Liang, Yuechen Zhang, Jiaya Jia

arXiv:2606.24033v1cs.LGcs.CL

TL;DR

Low-bit KV-cache quantizers often treat keys as flat vectors despite RoPE making their logit contributions block-structured. Block-GTQ allocates bits by RoPE-block energy and improves fidelity and long-context performance, while its packed serving path supports contexts where fp16 OOMs.

  • Problem

    Flat-vector KV quantization overlooks RoPE’s position-dependent two-dimensional block structure, despite the KV cache creating major long-context memory and bandwidth costs.

  • Method

    Block-GTQ computes label-free RoPE-block energy scores and greedily assigns integer key-cache bit widths under a fixed average-bit budget, using uniform TQ-MSE for values.

  • Results

    Block-GTQ cuts per-layer RoPE-logit MAE by 32–80% across ten models and wins all 367/367 layer comparisons against uniform TQ-MSE.

  • Takeaways & Limitations

    At K3V3, packed Block-GTQ enables 3.24× KV-cache compression, 1.34× faster 128K inference, and feasibility at 256K/512K where fp16 OOMs.

  • Takeaways & Limitations

    Block-GTQ allocates bits only on keys, leaves values uniform, and its fused decode path is an initial single-GPU implementation.

Abstract

from arXiv · show

Existing low-bit KV-cache quantizers often treat each cached key as a flat vector. Under RoPE, however, a key's contribution to a future attention logit decomposes into a position-dependent sum over two-dimensional frequency blocks. This makes key-cache quantization a block-wise bit-allocation problem: high-energy RoPE blocks are more sensitive to quantization error and should receive more bits. We introduce Block-GTQ, a RoPE-aware bit allocator for key-cache quantization built on TurboQuant-MSE(TQ-MSE). For each layer and KV head, Block-GTQ computes a label-free energy score for each RoPE block and greedily allocates integer bit widths by marginal gain. Under matched K/V bit budgets, Block-GTQ better preserves RoPE query-key logits on a ten-model diagnostic panel, cutting per-layer MAE by 32-80% at 2 and 3 b/dim K-only quantization and winning all 367/367 layer comparisons against uniform TQ-MSE. These fidelity gains translate to stronger downstream long-context retrieval, understanding, and reasoning. At K2V2 on Llama-3.1-8B-Instruct, Block-GTQ raises the six-task NIAH average from 70.6 to 97.4, and the LongBench-EN average from 36.87 to 53.31. On AIME 2024/2025 with DeepSeek-R1-Distill-Qwen-7B, without an fp16 recent-key buffer, Block-GTQ at K3V2 scores 51.7/37.5, close to fp16's 54.2/37.9, whereas uniform TQ-MSE collapses to 0.0/0.0. We further implement a packed-cache serving path. On a single H800 GPU with Qwen2.5-3B-Instruct, packed K3V3 achieves 3.24x KV-cache compression with fp16-comparable quality, runs 1.34x faster than fp16 FlashAttention2 at 128K context, reduces peak memory from 56.31 GB to 19.85 GB, and remains feasible at 256K and 512K where fp16 OOMs. Code is available at https://github.com/JIA-Lab-research/blockgtq.

1 Introduction

Block-GTQ reframes RoPE key-cache quantization as position-dependent logit preservation over two-dimensional frequency blocks, allocating more precision to blocks with greater influence. It combines label-free RoPE-block energy scores with TQ-MSE to greedily assign integer bit widths and improves logit fidelity and downstream long-context performance over uniform allocation.

  • Motivation: About 40 GiB at a 128K-token context is required by an fp16 KV cache in a GQA-style 70B-class model.The cache costs about 320 KiB per token in the cited 80-layer, 8-KV-head configuration.
  • RoPE-aware formulation: RoPE key logits decompose into a position-dependent sum over 2 × 2 frequency blocks, so cached keys are not used through a flat-vector interface.Each block contributes q^(i)⊤R^(∆θ_i)k^(i) to the future query-key logit.
  • Method: Block-GTQ computes label-free RoPE-block energy scores and greedily assigns integer bit widths under a fixed average-bit budget using the TQ-MSE 4−b error law.Blocks with the same assigned width are grouped and encoded with the original TQ-MSE local quantizer.
  • Results: 32–80% lower per-layer RoPE-logit MAE is achieved by Block-GTQ across models at K-only budgets, with 367/367 layer comparisons won against uniform TQ-MSE at each budget.The allocator targets uneven RoPE-block influence, where a few frequency blocks may carry most of the query-key signal.
  • Downstream impact: At K2V2 on Llama-3.1-8B-Instruct, the six-task NIAH average rises from 70.6 to 97.4 and the eight-task LongBench-EN average from 36.87 to 53.31.These results are reported relative to uniform-allocation TQ-MSE.
  • Downstream impact: At K3V2 on DeepSeek-R1-Distill-Qwen-7B, Block-GTQ scores 51.7/37.5 on AIME 2024/2025 without an fp16 recent-key buffer.The reported scores are close to fp16 performance.

2 RoPE-Structured Key-Cache Error

Under RoPE, query-key logit error decomposes independently across two-dimensional frequency blocks, with each block’s contribution controlled by query norm and key-error norm. This block-wise bound motivates integer bit allocation that assigns more bits to higher-energy blocks under a fixed budget.

  • RoPE-Structured Error: RoPE decomposes each query-key logit into a sum over independent two-dimensional frequency blocks.For relative offset Δ, each block is transformed by its own 2 × 2 rotation.
  • RoPE-Structured Error: Each block contributes independently to the error bound, with no cross-block terms.RoPE rotations preserve every query/key block’s ℓ2 norm, enabling energy statistics to be computed before RoPE.
  • Deployment Error: Key-cache compression perturbs attention logits through block-specific query norms and key-error norms.The analysis uses pre-RoPE coordinates and relative offsets even though deployment stores post-RoPE keys; queries are computed on the fly, motivating focus on keys.
  • Bit Allocation: Block-GTQ chooses integer bit widths for every RoPE block under per-layer, per-KV-head minimum, maximum, and total-budget constraints.The allocation keeps every RoPE block cached while enforcing b_min ≤ b_i ≤ b_max and a fixed sum of bit widths.
  • Bit Allocation: Higher-energy RoPE blocks receive more bits because their expected error-bound contribution combines block energy with the quantizer’s bit-dependent rate.The ideal block weight is s⋆_i = E[∥q^(i)∥₂ ∥k^(i)∥₂].

3 Block-GTQ: RoPE-Block Bit Allocation

Block-GTQ assigns bits across RoPE blocks using label-free energy scores based on marginal query/key second moments, then greedily spends each layer–KV-head budget where the next bit yields the largest objective reduction. The resulting schedule is implemented by grouping blocks with equal widths and encoding each group jointly with TQ-MSE.

  • Energy scoring: Block-GTQ uses an AM-GM-based energy score from marginal Q/K second moments, which may overestimate but never underestimates the paired-product score in expectation.Expectations are computed over a short unlabeled calibration prefix.
  • Greedy allocation: For each layer and KV head, Block-GTQ initializes every RoPE block at bmin and allocates remaining integer budget bits by largest current marginal reduction.Each added bit divides that block’s next marginal gain by four, prioritizing high-score blocks first.
  • Greedy allocation: Greedy allocation is optimal for the stated objective under positive scores and a feasible integer budget.The guarantee applies to the per-layer, per-KV-head allocation problem.
  • Physical cache layout: The scheduled blocks are grouped by assigned bit width, concatenated after RoPE, and encoded per group with one TQ-MSE encoder.Uniform TQ-MSE is the special case where all blocks share one rate.

4 Serving Block-GTQ from a Packed Cache

Block-GTQ serves attention directly from a packed HBM cache, decoding only the current time tile into kernel-local temporaries rather than materializing a full fp16 KV cache. This reduces per-step memory traffic through packed codes and norms while supporting layer-wise full-length prefill.

  • Packed-cache serving: The packed serving path stores K/V code streams, norms, and static layout metadata in HBM, while fused attention decodes only the current tile locally.Decoded fp16 K/V remain in kernel-local temporaries and are consumed directly in QK and PV products.
  • Packed-cache serving: The fused kernel unpacks tiles, dequantizes with an L1-resident shared fp16 codebook, rescales by norms, and computes QK⊤ and PV.Nibble extraction handles ≤4-bit groups, while higher-bit K groups use byte loads.
  • Memory-bandwidth efficiency: 157 B per token and KV head at K3V3 versus 512 B for an fp16 pair yields ∼3.26× lower per-step HBM traffic.Long contexts are split along the key axis and recombined with an exact log-sum-exp merge.
  • Memory-bandwidth efficiency: The packed path is marginally slower at short context because in-kernel unpacking adds a fixed per-step cost absent from fp16 FlashAttention-2.At longer contexts, the bandwidth reduction enables the packed path to overtake the fp16 baseline, as indicated by the passage’s continuation.
  • Prefill: Prefill populates the cache layer by layer in one full-length pass, using separate batched Triton kernels to quantize and pack all heads’ keys and values.The QKV/MLP projections and rotary embedding run as full-T matrix multiplications, with code streams and norms written directly to the cache layout.

5 Related Work

Prior work addresses long-context KV-cache costs through serving systems, context extension, quantization, and RoPE-aware compression. Block-GTQ differs by retaining every RoPE block while greedily assigning precision from a block-energy score.

  • Long-context inference and KV-cache memory: Long-context inference is constrained by repeatedly reading a KV cache that grows with sequence length, motivating cache-management and context-extension methods.PagedAttention and CacheGen manage and reuse KV state, while YaRN and LongLoRA modify how models reach longer windows.
  • KV-cache quantization: Most KV-cache quantizers optimize reconstruction or outlier objectives at channel, token, group, or vector granularity.Representative methods combine low-bit storage with outlier or mixed-precision adjustments, while others target sink tokens, outliers, or query-subspace structure.
  • RoPE-aware KV-cache quantization: RoPE-aware methods exploit structure before RoPE, through commuting codebooks, or through frequency selection, pair pruning, and geometric key-importance scores.Examples include KVQuant, RotateKV, CommVQ, EliteKV, RAP, and TriAttention.
  • Non-uniform precision allocation: Block-GTQ greedily allocates precision per RoPE block using a block-energy score derived from a RoPE logit-error bound, without dropping any block.This distinguishes it from approaches that select frequencies or prune RoPE-aligned pairs.

6 Experiments

Experiments show that Block-GTQ’s RoPE-aware mixed-rate allocation improves cache fidelity and downstream long-context retrieval, understanding, and reasoning over uniform TQ-MSE. A packed-cache implementation also delivers substantial memory savings and faster long-context decoding while remaining feasible beyond fp16’s memory limit.

  • RoPE-logit fidelity: At 3 b/dim K-only quantization, Block-GTQ reduces RoPE-logit MAE versus uniform TQ-MSE on all 10 models and wins 367/367 layer comparisons.Non-uniform RoPE-block energy profiles produce non-uniform bit allocations.
  • Calibration: The allocator’s rate law makes a misplaced bit at b=3 (K3V3) cost roughly 4× less than at b=2 (K2V2).Calibration was ablated across length and corpus, with additional prefix-noise sensitivity analysis.
  • Long-context retrieval: 98.4/96.8/97.4: Block-GTQ’s NIAH Overall (%) on Llama-3.1-8B-Instruct at K3V3/K3V2/K2V2 stays close to fp16’s 99.6 ceiling.At K2V2, TQ-MSE drops to 70.6, while KIVI-ScaleOnly never exceeds 35.4 Avg.
  • Long-context understanding: 59.08/58.84/53.31: Block-GTQ’s LongBench-EN Overall on Llama-3.1-8B-Instruct at K3V3/K3V2/K2V2 remains closest to fp16’s 59.83 ceiling.At K2V2, TQ-MSE scores 36.87 and KIVI-ScaleOnly scores 38.46.
  • Reasoning: 51.7/37.5 vs 54.2/37.9: without a recent-key fp16 buffer, Block-GTQ at K3V2 stays close to fp16 on AIME 2024/2025 for DeepSeek-R1-Distill-Qwen-7B.The evaluation reports average pass@1 over 8 samples per problem and compares no-buffer with protected-buffer regimes.
  • Packed-cache serving: 1.34× faster and 56.31 GB to 19.85 GB: at 128K context, packed Block-GTQ K3V3 beats fp16 FlashAttention2 while reducing peak memory.fp16 OOMs at T ≥256K, whereas Block-GTQ continues to run; uniform TQ-MSE is modestly faster but has collapsed quality.

7 Conclusion … A.1 Post-RoPE Cache and Pre-RoPE Coordinates

The paper concludes that Block-GTQ reframes RoPE-aware K-cache compression as block-level rate allocation, improving logit fidelity and enabling efficient long-context serving. The appendices provide theory, diagnostics, deployment data, and coordinate-level analysis, while identifying V allocation and serving scalability as future work.

  • 7 Conclusion: Block-GTQ assigns more bits to high-energy RoPE blocks using a label-free energy score, while encoding K and uniformly quantized V with TQ-MSE.RoPE attention decomposes exactly over two-dimensional frequency blocks, whose energies are non-uniform.
  • 7 Conclusion: 32–80% lower per-layer RoPE-logit MAE and 367/367 winning layer comparisons distinguish Block-GTQ from uniform TQ-MSE at 2 and 3 b/dim K-only budgets.These results hold across a diverse ten-model panel at both budgets.
  • 7 Conclusion: 3.24× KV-cache compression at K3V3 enables long-context inference with the packed-cache serving path beyond the context range reached by fp16 FlashAttention2.The deployment result is reported on a single H800.
  • 7 Conclusion: Block-GTQ allocates bits only on K, leaving V uniform; V-side allocation, joint K+V optimization, denser packing, multi-GPU serving, and batched serving remain open directions.The fused decode path is described as an initial single-GPU implementation.
  • Appendix Roadmap: Appendix A presents proofs, Appendix B documents diagnostics and allocation analyses, Appendix C studies calibration and stability, Appendix D describes long-context and reasoning protocols, and Appendix E reports deployment data.The roadmap covers error bounds, block weights, greedy optimality, cross-model perplexity, and footprint, latency, memory, and perplexity tables.
  • A Supplementary Theory Details: The supplementary theory centers on three facts: deployed K-cache error is a RoPE-logit error, that error has a per-block bound, and greedy allocation exactly optimizes the resulting objective.The appendix supplies auditability details behind the coordinate change, error bound, absolute-error chain, and block weight.
  • A.1 Post-RoPE Cache and Pre-RoPE Coordinates: The analysis can express post-RoPE cache behavior in pre-RoPE coordinates by applying the transpose of the absolute RoPE rotation to the decoded key.For a query at position n, this coordinate change supports the relative-position formulation.
  • A.1 Post-RoPE Cache and Pre-RoPE Coordinates: RoPE is orthogonal block by block, so the coordinate change preserves block norms and expresses deployed post-RoPE cache error as relative-position logit error.The transformation changes representation without changing block norms.

A.2 Proof of the Per-Block Accounting Bound … B.3 Per-Layer RoPE-Logit MAE

The appendix formalizes RoPE-aware per-block error accounting, derives the Block-GTQ weighting proxy and greedy optimality, and documents diagnostic coverage, allocation heterogeneity, and per-layer RoPE-logit MAE evaluation across diverse models.

  • A.2 Proof of the Per-Block Accounting Bound: Lemma 2 provides a per-block accounting framework for attention-logit error under RoPE.It considers a query position, cached-key position, and the equivalent pre-RoPE decoded key.
  • A.2 Proof of the Per-Block Accounting Bound: Triangle inequality and Cauchy–Schwarz supply the inequalities used to bound the per-block attention-logit error.
  • A.3 From the Block Bound to the RoPE-Block Weight: Block-GTQ uses RoPE-block energy scores with TQ-MSE bit-error decay as a rate-allocation proxy, not as a tight absolute-error consequence.The 4−bi factor comes from the local MSE-oriented quantizer, while the score reflects RoPE-logit sensitivity.
  • A.3 From the Block Bound to the RoPE-Block Weight: The 4−bi rate is consistent with squared logit-error decay, with the energy score serving as a simpler second-moment proxy for the product weight.This follows by squaring the per-block bound and applying Cauchy–Schwarz.
  • A.4 Proof of Greedy Allocation Optimality: Greedy allocation is proved optimal by representing each feasible allocation as a prefix-constrained selection of exactly K geometrically decreasing marginal gains.The exchange proof preserves an optimal feasible set containing the greedy prefix by adding the next largest available gain.
  • B.1 Model Panel and Activation Extraction: The attention diagnostics cover a ten-model panel selected for architectural coverage, including nine GQA models and one MLA model.The panel spans Qwen, Llama, DeepSeek-R1, Mistral-Nemo, GLM-4-9B, and DS-V2-Lite architectures.
  • B.2 Bit Allocation across Models: Every model produces non-uniform Block-GTQ allocations at 3 b/dim and 2 b/dim, while every layer uses multiple bit widths at 3 b/dim.Across models, the mean number of distinct bit levels per layer is grps ∈[4.0, 5.6], and entropy curves typically lie near H ∈[1.3, 1.6].
  • B.3 Per-Layer RoPE-Logit MAE: Per-layer RoPE-logit MAE averages clean-versus-quantized key error across KV heads using analytically matched RoPE rotations, with lower values preferred in Table 11.The protocol handles MLA’s single shared decoupled RoPE key and GLM-4-9B’s partial rotary dimensions explicitly.

B.4 Attention Diagnostics across Models · C Calibration Robustness · C.1 Calibration length ablation

Across ten models, Block-GTQ improves RoPE-aware attention fidelity over uniform TQ-MSE, with advantages depending on available RoPE structure. Calibration length affects K2V2 more than K3V3, while the allocator uses GQA-aware per-block energy scores.

  • B.4 Attention Diagnostics across Models: The diagnostics use no recent-key buffer, isolating the K quantizer; KIVI is represented by buffer-free KIVI-ScaleOnly, while V remains fp16.Softmax KL weights errors by fp16 attention mass, and top-10 overlap measures shared membership among the ten most-attended tokens.
  • B.4 Attention Diagnostics across Models: Block-GTQ jointly achieves the lowest softmax KL and highest top-10 overlap on 7/10, 8/10, and 9/10 models at 2, 3, and 4 b/dim.Relative to TQ-MSE, panel-mean softmax KL drops by 3.28× / 3.88× / 4.63×, while top-10 overlap rises by 12.8 / 8.9 / 5.6 percentage points.
  • B.4 Attention Diagnostics across Models: Block-GTQ beats TQ-MSE on every model–bit-budget cell for both diagnostics and beats KIVI-ScaleOnly on eight of ten models.The remaining architectures have only 32 RoPE-carrying blocks, versus 64 in standard GQA models, reducing the structure available for allocation.
  • C Calibration Robustness: Block-GTQ computes each layer- and KV-head-specific allocation from per-RoPE-block energy scores over a short calibration prefix, with calibration choices affecting prefix length, score function, and corpus.Under GQA, the Q-side score averages squared norms across the query heads served by each KV head rather than squaring their averaged vector.
  • C.1 Calibration length ablation: At K2V2, N = 64 reaches 95.68 while only N = 2048 wins cleanly at 97.36, showing non-monotone calibration-length sensitivity.The m-query subtask ranges from 74.24 at N = 1024 to 86.70 at N = 2048, while binary subtasks remain ≥91.92.
  • C.1 Calibration length ablation: At K3V3, every calibration length lies within 1.07 percentage points of the N = 2048 overall result, and m-query stays within 3.20 points.Table 14 reports that every K3V3 subtask is within approximately 3.54 points of the N = 2048 baseline.

C.2 Energy score ablation

The energy-score ablation compares five Block-GTQ scoring functions under identical calibration and allocation settings, evaluating them with NIAH at the rate-sensitive K2V2 budget on Llama-3.1-8B-Instruct.

  • Energy score ablation: Five energy scores are compared: the default qk_avg, symmetric qk_max and qk_product, and single-sided k_only and q_only variants.The single-sided variants isolate whether the key or query side carries the attention signal.
  • Energy score ablation: All five variants use the same calibration—the first 2048 tokens of WikiText-2 test—and the same Block-GTQ allocator.This controls calibration and allocation while varying only the energy-score function.
  • Energy score ablation: NIAH is run on Llama-3.1-8B-Instruct at K2V2, where the energy-score choice is most consequential.The results are reported as per-task NIAH pass rates in Table 15.

C.3 Calibration corpus ablation … D.1.2 LongBench-EN Protocol

Block-GTQ’s calibration and evaluation protocols show that calibration-domain mismatch matters most at lower bit budgets, while downstream long-context tests use paired, task-specific measurements across NIAH and LongBench-EN. The protocols specify calibration sources, allocation-stability diagnostics, retrieval subtasks, scoring metrics, and inference constraints.

  • C.3 Calibration corpus ablation: 3.70 pp: At K2V2, code calibration lowers Overall NIAH performance versus WikiText-2, while PG19 stays within 0.28 pp and C4 drops 2.78 pp.At K3V3, all four calibration corpora remain within 0.34 pp, indicating lower sensitivity at the higher budget.
  • C.3 Calibration corpus ablation: 11–16×: K3V3 reduces the observed NIAH swing from calibration mismatch through the 4−b rate law, whose per-misplaced-bit penalty is about four times smaller than at K2V2.The allocator can misplace bits when off-domain calibration changes per-block energy rankings.
  • C.4 Cross-model PPL and allocation-distance diagnostics: 1.4–2.7×: At K3V3 with Ncal = 128, Hamming distance exceeds the within-source noise floor, HB@4 falls by 10–13 pp, and regret remains 1.2–3.1%.The reference noise floor is Hamming 0.07–0.08, HB@4 0.86–0.93, and regret +0.4–0.9% for Ncal = 2048 seeds.
  • D Downstream Evaluation Details: 2048 tokens: Block-GTQ calibration uses WikiText-2 test prefixes, while TQ-MSE is data-independent and KIVI-ScaleOnly follows Appendix B.4.The calibration protocol is shared between NIAH and LongBench-EN.
  • D.1 Long-Context Tasks: 6 NIAH variants: The protocol inserts synthetic key–value needles into filler haystacks and tests basic, distractor-resistant, selective, multi-key, multi-value, and multi-query retrieval.The first three tasks use binary scoring; the last three score the fraction of correct responses.
  • D.1.1 NIAH Protocol: 4K–128K: Each NIAH task, context length, and needle-depth cell averages three haystack samples across eleven depths, with paired needle sets shared across methods and budgets.Needle depths span 0%, 10%, …, 100%.
  • D.1.2 LongBench-EN Protocol: 8 subtasks: LongBench-EN evaluates single- and multi-document QA, summarization, few-shot classification, synthetic retrieval, and code completion using QA-F1, ROUGE-L, accuracy, retrieval score, and edit similarity.Each task uses a LongBench-defined output-token cap.
  • D.1.2 LongBench-EN Protocol: 31,500 tokens: LongBench-EN inputs are middle-truncated, decoded greedily, and scored by the unweighted mean across eight subtasks in the Avg column.Per-task output caps are applied during inference.

D.2 Reasoning Tasks (AIME)

This section specifies the AIME evaluation protocol and contrasts protected-buffer and no-buffer decoding regimes. It also documents the quantization and calibration configurations used for each method.

  • Protocol: All AIME runs use K3V2 budgets, stochastic decoding at temperature 0.6 and top-p 0.95, eight samples per problem, and report pass@1 (avg@8).Generation is capped at 32,768 tokens; PM-KVQ and KIVI use their official schemes, with KIVI-ScaleOnly in the no-buffer regime.
  • Buffer configurations: The protected-buffer regime retains a 4-token fp16 sink and 128-token fp16 recent window, applying the same allowance to buffer-free TQ-MSE and Block-GTQ.The 128-token recent span follows PM-KVQ’s protected configuration, while the 4-token sink standardizes attention-sink handling across methods.
  • Buffer configurations: The no-buffer regime sets sink and recent windows to zero, forcing every attended token through the compressed cache.KIVI-ScaleOnly uses per-channel K quantization with a 32-token rolling fp32 statistics buffer, while V uses TQ-MSE.
  • Buffer configurations: In the no-buffer regime, PM-KVQ progressively reduces layer precision from 16→8→4→2 bits as sequence length exceeds calibrated per-layer memory budgets.Each K/V is quantized on arrival with per-group 128-channel asymmetric quantization; TQ-MSE and Block-GTQ operate without buffers.
  • Calibration: Block-GTQ calibration uses the first 2048 WikiText-2 test-split tokens, while TQ-MSE is data-independent.The Block-GTQ energy-score and bit-allocation procedure is specified in Appendix C.

E Deployment Protocol and Extended Results … E.3 Perplexity at Long Context

The appendix details a single-H800 deployment protocol for Qwen2.5-3B-Instruct, including allocated footprint accounting, packed-cache latency and memory comparisons, and leakage-free long-context perplexity evaluation. Block-GTQ preserves perplexity close to fp16 through 128K, while uniform TQ-MSE is dramatically worse at every tested context.

  • E Deployment Protocol and Extended Results: The deployment benchmarks run Qwen2.5-3B-Instruct on one H800 80GB GPU against uniform TQ-MSE and fp16 FlashAttention-2, using a 64-token WikiText-2 calibration prefix.Decode latency is the median over 20 timed autoregressive steps, and peak memory includes model weights and runtime components.
  • E.1 Allocated Footprint Accounting: At D = 128, ideal pure 3-bit K+V storage provides 5.33× fp16 compression, whereas deployed K3V3 allocates approximately 157 bytes per token and KV head.The approximately 61-byte overhead is dominated by nibble- or byte-rounded per-coordinate storage and other deployment metadata.
  • E.2 Decode Latency and Memory: Three-Way Comparison: The packed-cache paths share one interface: uniform TQ-MSE uses a uniform 3-bit K budget, while Block-GTQ assigns K bits by RoPE block.The evaluation reports per-step decode latency, prefill time, speedup, KV footprint, compression, and peak GPU memory across five context lengths.
  • E.2 Decode Latency and Memory: Three-Way Comparison: 1.34× faster than fp16 FA-2, Block-GTQ decodes at 128K context; uniform TQ-MSE reaches 1.56×, while fp16 is fastest at contexts up to 64K.At longer contexts, fp16 runs out of memory because peak total memory exceeds 80 GB.
  • E.2 Decode Latency and Memory: Three-Way Comparison: Approximately 3.4× KV-footprint reduction is achieved by both compressed paths under K3V3 relative to fp16, with peak-memory curves nearly overlapping.Block-GTQ uses slightly more resident KV memory, and their peak-memory difference reaches at most 0.98 GB at 512K.
  • E.3 Perplexity at Long Context: The perplexity protocol scores the same next 1000 WikiText-2 tokens after T input tokens, with non-overlapping 64-token calibration and evaluation windows.Using the same scored positions for all methods and a larger nppl reduces token-averaging noise while avoiding calibration-evaluation leakage.
  • E.3 Perplexity at Long Context: Within 128K, Block-GTQ’s perplexity remains 1.6%–3.6% above fp16, while uniform TQ-MSE is 14× to 36,879× higher than Block-GTQ at every context.At 256K and 512K, fp16 OOMs and both packed paths degrade sharply because the model exceeds its supported context, not because of cache compression.
Loading 2606.24033v1…