PaperlayerView exact revision on arXiv

Source-linked AI summary

SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference

Daeha Lee, Do-Hyung Kim, Jae-Hong Kim

arXiv:2608.28911v1cs.LGcs.CLcs.IT

TL;DR

Long-context inference is limited by KV-cache memory, and the paper asks whether quantization can reduce this cost without graceful-quality failures or token deletion. It maps the quality cliff, then uses all-token-preserving, importance-ranked mixed precision to interpolate above it. SemKV reaches 6.0× storage reduction at 2.39 bits with no statistically detectable quality difference from full KV, while TurboQuant-MSE raises the operating point to 7.9×.

  • Problem

    KV-cache memory grows linearly with context length, motivating evidence about where uniform quantization fails and how compression can preserve semantic information without deleting tokens.

  • Method

    SemKV preserves every token, ranks tokens using model-internal semantic importance, and assigns two adjacent above-cliff precisions, with a deferred block-exact extension for generation-time tokens.

  • Results

    6.0× KV-cache storage reduction versus FP16 at 2.39 average bits is statistically indistinguishable from full KV, while TurboQuant-MSE raises the no-detectable-loss operating point to 7.9×.

  • Takeaways & Limitations

    The supported recipe is to measure the quality cliff for the target deployment setting and interpolate above it with mixed precision.

  • Takeaways & Limitations

    The measured cliff varies with context-budget composition, evaluation protocol, base quantizer, seed, and task; stress evidence for indicator separation is based on a small below-cliff sample.

Abstract

from arXiv · show

The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length. We show that uniform KV quantization on a fractional-bit grid does not degrade gracefully: under a prespecified multi-seed statistical protocol, Llama-3.1-8B-Instruct with an affine quantizer is statistically indistinguishable from FP16 KV down to 2.322 code bits/value and collapses at 2.0 bits - a quality cliff in (2.0, 2.322] that reappears in generation-time quantization and multi-turn dialogue and transfers to Mistral-7B. The cliff reframes importance-aware mixed precision: above it, eight model-internal importance indicators are statistically interchangeable, so the benefit of mixing is grid interpolation, reaching average precisions uniform quantization cannot realize. SemKV preserves every token, ranks tokens by a model-internal score, and assigns two adjacent above-cliff precisions, achieving a measured 6.0x storage reduction with no statistically detectable quality difference from full KV (n=900, three seeds), and outperforming FP16 token pruning granted a 1.5x larger memory budget. Replacing the affine base with a distortion-optimized quantizer (TurboQuant-MSE) lowers the cliff in every protocol tested, raising the no-detectable-loss operating point to 7.9x. The recipe: measure the cliff for the target deployment setting, then interpolate above it.

1. Introduction

Long-context inference makes KV-cache memory a central bottleneck, while uniform quantization exhibits a sharp quality cliff rather than graceful degradation. SemKV preserves every token and uses above-cliff mixed precision to interpolate otherwise unreachable operating points.

  • KV-cache memory grows with context length and constrains feasible batch size and context budget in practical serving systems.
  • Uniform quantization remains statistically indistinguishable from full-precision KV from 3.0 down to 2.322 bits, then collapses in (2.0, 2.322].The same affine-base collapse reappears in single-turn QA, generation-token quantization, multi-turn dialogue, and Mistral-7B.
  • SemKV preserves every token and assigns two above-cliff precisions by model-internal importance ranking, reaching average precisions uniform quantization cannot realize.At 2.39 bits, it is statistically indistinguishable from full KV.
  • 6.0× KV-storage reduction versus FP16 is achieved at the 2.39-bit operating point with no statistically detectable deficit from full KV.The measurement includes metadata and uses all-token-preserving mixed precision.
  • FP16 pruning collapses to 0.065–0.149 under matched adverse memory budgets, whereas SemKV at 2.39 bits remains statistically indistinguishable from full KV at 0.438.The controlled comparison implicates token deletion as the dominant source of failure.
  • TurboQuant-MSE shifts the collapse boundary down to (1.585, 2.0] in prefill and multi-turn and to (1.0, 1.585] for generation, enabling a 7.9× operating point.SemKV’s token-axis allocation composes unchanged with the alternative quantizer.

2. Related Work

Prior KV-cache compression methods use uniform quantization, token removal, positional or attention-pattern selection, or broader memory and attention optimizations. SemKV is positioned as an all-token-preserving, token-wise fractional-grid method with generation-time support.

  • Importance-aware KV methods allocate precision or retain tokens using signals such as eviction policies, attention scores, recency, or sensitivity.
  • SemKV preserves all tokens and adjusts only their precision according to importance, unlike pruning or sparse selection methods that remove tokens or restrict accessibility.
  • Attention and positional methods improve computation or positional generalization, whereas SemKV directly compresses KV-cache memory during decoding.
  • SemKV extends mixed-precision inference to the token dimension of the KV cache.
  • Uniform KV quantization applies one bit-width to all tokens, while eviction and pruning remove tokens considered less important.
  • Table 1 distinguishes SemKV by combining all-token preservation, token-wise fractional-grid precision mapped from a measured cliff, and generation-time support.

3. SemKV Method

SemKV reduces KV-cache memory without token pruning by ranking tokens using model-internal semantic importance and assigning them mixed precision before decoding.

  • SemKV generates a full-precision KV cache during prefill, estimates token-level semantic importance, and separates tokens by importance ranking.
  • SemKV retains all token positions while reducing KV-cache memory through token-wise bit allocation.
  • The method constructs a mixed-precision KV cache by assigning higher precision to more important tokens before decoding.

3.2. Problem Formulation

The formulation represents KV-cache storage as scaling with sequence length and precision, while SemKV keeps sequence length fixed and varies precision across tokens.

  • KV-cache storage is approximated as MKV = 2 · L · HKV · T · D · b bits for one sequence, ignoring quantization metadata.Here L is the number of layers, HKV the number of KV heads, T the sequence length, D the head dimension, and b the precision.
  • SemKV keeps sequence length T unchanged while assigning each token a different bit-width bi.
  • Semantic importance is estimated from model-internal signals, with late-layer hidden-state dynamics used by default as an importance proxy.

3.3. Semantic Importance Estimation

SemKV estimates token importance from model-internal signals and uses within-prompt rankings to guide allocation. Its default indicator measures hidden-state changes across the final Transformer layers, while ablations cover multiple signal families.

  • SemKV computes token importance from model-internal indicators and uses only within-prompt rankings, so score normalization is unnecessary.The scoring pipeline can use hidden-state, KV-norm, logit, or attention signals.
  • The default semantic score averages hidden-state changes across the last m Transformer layers.Tokens with larger representation changes are treated as more important under the default delta-based indicator.
  • SemKV’s indicator families include hidden-state dynamics, final-layer magnitude, cosine change, KV norms, self-information, and accumulated attention.The attention family requires materializing attention weights, unlike the other listed families.
  • The allocation procedure remains identical across indicators because scores affect only the ranking of tokens within each prompt.A random selector provides the uninformative baseline in the ablations.

3.4. Token-Wise Mixed-Precision Allocation

SemKV first measures the quality cliff for a fixed deployment setting, then assigns two above-cliff precisions according to token-importance rankings. This produces mixed precision by interpolating between adjacent grid points without estimating a score threshold.

  • SemKV determines the precision pair (bhigh, blow) through offline cliff mapping for each fixed deployment setting.The mapping sweeps uniform quantization levels against full precision and examines adjacent grid-point differences.
  • Above the measured cliff, SemKV interpolates between adjacent grid points that uniform quantization cannot realize directly.The above-cliff region is generally flat, although multi-turn evaluation can retain small residual deficits.
  • SemKV ranks tokens by importance, assigns higher precision to the top-ranked fraction, and assigns lower precision to the remaining tokens.The allocation retains all token positions while reducing average KV precision.
  • The allocation uses only the top-r ranking, with no estimated score threshold.The precision pair should be revalidated when the model, quantizer, protocol, or context-budget composition changes materially.

3.5. KV Cache Quantization

SemKV applies token-level bit assignments to both key and value tensors to form the quantized KV cache used for decoding. The same token-wise assignment is broadcast across all layers and KV heads rather than allocated along the head dimension.

  • SemKV quantizes each token’s key and value tensors using its assigned bit width.The resulting tensors form the mixed-precision KV cache used during decoding.
  • Token-wise precision assignments are broadcast across all layers and KV heads.SemKV never allocates bits along the head dimension itself.
  • Experiments use fractional-level precision pairs such as 2.585/2.322 bits.

3.6. All-Token-Preserving Property

SemKV preserves the complete input-token set in the KV cache and changes only the precision assigned to each token. This distinguishes it from pruning methods that retain only a subset of tokens.

  • SemKV preserves every input token in the KV cache rather than retaining only a proper subset.This is the method’s all-token-preserving property.
  • SemKV changes token precision without deleting token positions.

3.7. Generation-Token Mixed-Precision Quantization

SemKV handles generated tokens with deferred, block-exact mixed-precision quantization. It accumulates scores, selects the prescribed high-precision fraction within completed blocks, and keeps the newest partial block in FP16 during decoding.

  • Generated-token scores are accumulated and quantized block-exactly in deferred fashion, meeting the prescribed high-precision fraction without threshold estimation.The procedure quantizes completed blocks after enough scores have accumulated.
  • The most recent partial block remains in FP16 during decoding and is quantized at each generation turn’s end using the same block-wise top-r rule.

3.8. SemKV Algorithm

SemKV constructs a mixed-precision KV cache by scoring prompt tokens, assigning higher precision to the top-ranked fraction, and quantizing the remaining tokens at lower precision. Generated tokens are handled separately by a block-exact procedure.

  • Prefill: SemKV’s prefill algorithm computes semantic importance scores using the last m layers and selects the top ⌊rT⌋ prompt tokens for high precision.
  • Prefill: Selected prompt tokens receive b_high while all other tokens receive b_low, after which their key and value states are quantized for decoding.
  • Decoding: The quantized prompt KV cache is used for decoding, while generated tokens are processed by the block-exact quantization algorithm.
  • Generated tokens: For each completed block, exactly round(rB) highest-scoring tokens receive b_high and the remainder receive b_low before in-place quantization.
  • Generated tokens: During generation, new tokens are appended at full precision and scored without making per-token bit-width decisions.
  • Generated tokens: The most recent partial block stays at full precision during decoding and is flushed with the same top-r rule at the end of each generation turn.

4. Experiments and Results

Across prespecified multi-seed experiments, SemKV identifies a sharp uniform-quantization quality cliff, interpolates safely between above-cliff precisions, and preserves quality while substantially reducing KV storage. The results extend across generation, multi-turn dialogue, model transfer, and indicator-cost trade-offs, with important protocol-specific residual floors.

  • Uniform quantization cliff: All uniform settings from 2.322 to 3.0 bits were statistically indistinguishable from full KV, while 2.0 bits collapsed on Llama-3.1-8B.The 2.0-bit result was Δ=−0.0448, p<10−7, whereas the higher-bit settings had |Δ|≤0.0094 and p≥0.09.
  • Grid interpolation: SemKV achieved no statistically detectable deficit from full KV at 2.39 average bits by allocating 25% of tokens 2.585 bits and 75% 2.322 bits.This above-cliff interpolation preserves every token and reaches an average precision unavailable to uniform quantization.
  • Indicator behavior: At the primary above-cliff operating point, all eight indicators were statistically interchangeable, but widened-gap below-cliff stress made indicator choice decisive.In the stress setting, indicators separated into a performance spectrum, and some performed below random selection.
  • Generation-time quantization: Generation-time quantization reproduced the cliff: 2.322-bit generation was statistically indistinguishable from FP16, whereas any mixed or uniform configuration with a 2.0-bit low side degraded significantly.The below-cliff configurations had Δ between −0.0127 and −0.0157, all p<10−3.
  • Scoring cost: Hidden-state scoring was O(1) per decoding step and nearly flat with context length, whereas attention recomputation was O(T) and reached 8.3× the hidden-state cost at 32k tokens.The hidden-state microbenchmark increased 3.1% from 1k to 32k tokens; attention scoring increased to 3.2× its 1k cost.
  • Memory footprint: 6.0× less storage than FP16 KV was measured for SemKV across 4k–128k contexts, outperforming 25%-token FP16 pruning while using a 1.5× smaller footprint.At 128k tokens, measured storage was 16.0 GiB for FP16 versus 2.65 GiB for SemKV.
  • Multi-turn dialogue: Multi-turn dialogue retained a 2.0-bit collapse and a small above-cliff residual floor, while SemKV at 2.41 average bits improved over uniform 2.322-bit quality but did not eliminate that floor.The SemKV deficit was −0.008 to −0.009, with p=0.03–0.06 depending on the run.

5. Discussion

The paper argues that mixed precision is most useful above the quality cliff, where preserving every token and interpolating between grid points provides the main benefit. It also identifies practical scope boundaries: the cliff varies with deployment conditions, and the reported efficiency comparisons are not universal benchmarks.

  • What mixed precision is for: Above the quality cliff, SemKV’s main benefit is interpolating between uniform quantization grid points while preserving every token.The paper contrasts this with regimes at or below the cliff, where indicator quality becomes decisive.
  • What mixed precision is for: A better base quantizer lowers the cliff and increases the compression available above it.The paper therefore treats the cliff as a deployment-specific measurement rather than a fixed model constant.
  • Deletion versus precision redistribution: Low-precision preservation succeeds where token deletion fails, even when pruning receives a larger memory budget.The controlled comparison attributes the difference to preserving the attention graph rather than selecting which tokens to delete.
  • Limitations: The cliff varies with context-budget composition, evaluation protocol, base quantizer, seed, and task, while the below-cliff indicator-separation evidence is limited.These factors constrain how broadly the reported operating points should be generalized.
  • Limitations: The pruning comparison is a structural control, and scoring-cost and storage measurements are microbenchmarks tied to one GPU class and a specific packed layout.The TurboQuant transfer also remains untested on other backbones.

6. Conclusion

SemKV uses an empirical quality-cliff map to guide all-token-preserving mixed-precision KV compression. The reported results show strong compression without detectable loss in several tested settings, while retaining explicit protocol and quantizer scope boundaries.

  • Conclusion: SemKV preserves all tokens and uses a fractional-bit quality-cliff map to guide mixed-precision KV compression.Its operating rule is to interpolate strictly above the measured cliff.
  • Conclusion: 6.0× KV-cache storage reduction versus FP16 was measured at 2.39 average bits on Llama-3.1-8B, with no statistically detectable LongBench deficit across eight indicators.Metadata is included in the reported storage reduction.
  • Conclusion: The approach transfers to generation-time tokens, multi-turn dialogue, and Mistral-7B, while multi-turn retains a small above-cliff floor.The controlled pruning comparison also favors low-precision preservation at a larger pruning memory budget.
Loading 2608.28911v1…