PaperlayerView exact revision on arXiv

Source-linked AI summary

Squeezing the Cache, Preserving the Truth: Monotonic Equipotential Allocation with Geodesia-KV

Vincenzo Dentamaro, Pancrazio Auteri, Giuseppe Pirlo

arXiv:2608.23599v1cs.ARcs.CL

TL;DR

KV-cache evaluations can conflate resident capacity with per-query read traffic and can be distorted by chunked teacher-forcing. Geodesia-KV addresses these issues with causal, monotonic precision allocation and rate-distortion-based compression, achieving strong retrieval and VRAM results while exposing scaling limitations in exploratory evaluations.

  • Problem

    KV-cache compression has separate resident-memory and per-query read-traffic demands, while chunked teacher-forcing can alter measured resident rates.

  • Method

    Geodesia-KV uses a graded precision ladder with causal block-wise allocation, rate-distortion configurations, and separated resident-memory and read-traffic accounting.

  • Results

    Geodesia-KV achieves 100.0% long-context retrieval accuracy and up to 84.2% VRAM reduction, including at 1,000,000 context tokens.

  • Takeaways & Limitations

    The vLLM integration demonstrates feasibility for memory-efficient long-context serving, including 1M-token inference on a single 16 GiB GPU.

  • Takeaways & Limitations

    Exploratory scaling results require progressive recomputation because the reported JSON results precede the evaluation-mode flag and use Q = 64.

Abstract

from arXiv · show

Current assessment of KV-cache compression performance confuses resident bits with read bandwidth and is affected by the artifacts of chunked teacher-forcing. We present Geodesia-KV, a family of training-free KV cache policies based on monotonic block-wise precision allocation, exact rate-distortion residuals, and query-sparse reading, enabling proper hardware-ready compression. With proper separation of resident and read bits and causal evaluation, we show that Geodesia-KV significantly outperforms other approaches. Specifically, on WikiText-2 with 16k context, the 5-bit operating point of Geodesia-KV results in lower perplexity at lower bitrate than KIVI-4 on Qwen. In addition, our compressed-Quest version delivers improved perplexity and reduces resident (9.83 vs 16.25 bits/value) and read rates (1.95 vs 2.32 bits/value) over baseline sparse methods on PG-19. As Geodesia-KV is implemented as native GeodesiaKVCacheManager plug-in of vLLM, Geodesia-KV fully removes the need for dense cache residency via monotonic bit demotion. With the full consumer hardware evaluation, Geodesia-KV leads to 1M-token context generation on a single 16 GiB GPU with up to 71.7% peak VRAM savings on all leading architectures (Qwen, Llama, DeepSeek).

1 Introduction

Geodesia-KV addresses evaluation errors in KV-cache compression by separating resident memory from read traffic and enforcing causal, Q=1 assessment. Its training-free policies combine monotonic precision allocation, rate-distortion residuals, and query-sparse reading with native serving integration.

  • Motivation: Resident cache capacity and per-query read volume are distinct resources, but sparse-method reporting can conflate them.Autoregressive KV memory grows linearly with context length, while query-sparse methods primarily reduce cache reads.
  • Motivation: 5.03 bits/value for a 64-token chunk versus 5.69 for a 1024-token chunk shows why chunked teacher-forcing breaks rate matching.Different query lengths can expose different current, unquantized regions to the cache policy.
  • Approach: Geodesia-KV maintains causality while applying a graded precision ladder across the entire context.The implementation also separates resident memory from read traffic and avoids future-query leakage and duplicated GQA allocation.
  • Evaluation and systems: Native vLLM integration and incremental experiments cover multiple checkpoints, corpora, architectures, and consumer-hardware context scaling.The contributions include regression tests, negative-result reporting, and a single 16 GiB accelerator demonstration.
  • Approach: Exact rate-distortion residuals select prompt-salient corrections, while Compressed-Quest combines graded K/V storage with query-sparse reading.The residual is target-free and the sparse branch retains exact key boxes for page ranking.

2 Related Work

Related work spans token eviction, sparse reading, KV quantization or merging, and architectural alternatives. Geodesia-KV differs by combining training-free scalar and centroid representations under one rate limit while evaluating resident and read costs separately.

  • Eviction and retained-token techniques: Eviction methods retain sinks, recent windows, heavy hitters, or layer/head budgets, but evicted tokens cannot answer later queries.Examples include StreamingLLM, H2O, SnapKV, PyramidKV, PyramidInfer, CAKE, and Ada-KV.
  • Query-sparse attention: Query-sparse methods rank or retrieve selected pages using bounds, representative tokens, structured sparsity, or hierarchical pruning.Their resident backing store and query volume require separate accounting.
  • KV quantization and merging: KV-compression methods use quantization, sparse corrections, low-rank structure, token merging, or cross-layer similarity.Geodesia-KV instead combines scalar quantization and a centroid state under a single rate limit, then optionally adds an irregular exact residual.
  • Architectural and systems alternatives: Architectural alternatives modify training or model structure, whereas Geodesia-KV is training-free and targets frozen Qwen checkpoints.Its prototype uses FlashAttention’s online-softmax recurrence, while GPTQ applies only to exploratory weights.
  • Long-context evaluation: WikiText-2 and PG-19 provide controlled language-model-loss evaluation, but the absence of RULER and LongBench remains a limitation.Perplexity alone does not demonstrate retrieval and downstream-task performance.

3 Method

Geodesia-KV allocates heterogeneous precision to causal KV blocks, adds target-free token residuals, and supports Compressed-Quest sparse reads. Monotonic demotion enables in-place compression, while an analytical rate-distortion allocator links attention mass and reconstruction error to bit depth.

  • Causal observation: EMA attention mass keeps allocation state strictly causal during autoregressive decoding.Using future queries to demote blocks would leak teacher-forced information.
  • Graded allocation: 64-token blocks use levels {16, 8, 4, 2, 1}, representing exact, scalar-quantized, or centroid KV representations.The 1-bit level corresponds to logically repeated KV centroids.
  • Graded allocation: Demotion is monotonic and irreversible after a block leaves the recent window, enabling in-place compression without a dense fallback copy.The allowed path includes demotion from 16 down to 4.
  • Lagrangian allocation: The allocator uses a Lagrangian relaxation to decouple per-block decisions and computes bit depth analytically from adaptive error tolerance.Higher attention mass receives higher bit depth, then continuous values are clamped to discrete ladder levels.
  • Rate-distortion token saliency: Token residual scores combine attention mass with squared distance from the block value centroid, selecting a fixed exact fraction from prompt activations.Selected high-score tokens are stored in 16 bits without target queries.
  • Compressed-Quest: Compressed-Quest stores graded K/V pages plus exact key summaries, ranking pages by box bounds while always reading the current partial page.The read cost includes summaries and graded K/V from selected and current pages.
  • Attention error certificate: The attention-error certificate is validated for graded reconstruction but is generally loose for hard sparse masking.Validation with dense references observed no violations.
  • Systems implementation: The packed attention prototype dequantizes inside an online-softmax loop, batches KV heads, and matches a PyTorch reference within 10^-4 relative tolerance.The native vLLM implementation removes dense tensors and supports incremental append.

4 Experimental Protocol

The protocol separates resident capacity from per-query read traffic and evaluates cache policies causally with one query per target token. It also uses paired, cluster-bootstrap uncertainty estimates and shared implementations for controlled comparisons.

  • Models and data: Primary experiments use Qwen2.5-3B-Instruct and Qwen3-8B in 16,384-token contexts on an NVIDIA RTX A6000.Activations use bfloat16; the 30B exploratory checkpoint is separate from the primary protocol.
  • Models and data: PG-19 evaluation exhausts the preregistered eleven-book window list rather than relying on the earlier six-book subset.The fifth book is excluded because it contains only 6,679 tokens.
  • Causal evaluation: Each window is evaluated by prefilling a precise 16k prefix, issuing one query per teacher-forced target, appending that token, and scoring the following-token NLL.Primary results use Q = 1 and 64 target tokens per window.
  • Statistical analysis: Perplexity comparisons use paired target tokens and map confidence intervals on mean NLL differences directly to perplexity ratios.Window-level cluster bootstrap resampling uses 20,000 draws and reflects the small number of clusters in interval width.
  • Metrics and baselines: The reported metrics include resident bits per K/V value, bits read per value per query, teacher-forced perplexity, attention KL divergence, cache size, and attention-only latency.Resident and read rates are distinct quantities in the protocol.
  • Metrics and baselines: Baselines are local semantic ports on a shared attention path, not executions of the original version-pinned kernels.This supports causal comparison but is not a substitute for an official code benchmark.

5 Results

Geodesia-KV shows small local Pareto improvements and a statistically supported compressed-Quest advantage on the full PG-19 test. However, several five-bit and table-level comparisons remain too uncertain for superiority claims.

  • WikiText-2 results: 0.0026 PPL at 3B and 0.0408 PPL at 8B are the reported local margins, corresponding to 0.04% and 0.39%.Associated KL values are directionally lower for Geodesia-KV than KIVI-4 at both model sizes.
  • Quest and compressed-Quest: 39.5% faster resident rate and 15.4% faster read rate accompany compressed-Quest’s significant PG-19 perplexity comparison against Quest.The paired ratio is 1.00516 with a 95% confidence interval of [1.00076, 1.00945].
  • Pareto frontier: 4.98 BPV is the point where Geodesia-KV matches full-KV while beating StreamingLLM and Quest at low resident rates.The figure describes this as the optimal Pareto frontier among the rates.
  • Frozen cross-corpus test: 0.10% is the full eleven-book PG-19 margin, with 5/11 winning windows and a paired 95% confidence interval that includes one.The earlier six-book subset produced a 0.69% loss and a 2/6 win count.
  • Statistical interpretation: The six-book subset was insufficient: its 0.69% five-bit result became an unmeasurable difference at 4.989 versus 5.031 bits/value on the full test.The resulting interval’s half-length was 0.9%, larger than the effect.
  • Rate-distortion residual: 4.985 bits/value with PPL 24.222 is the corrected graded-control result, while KIVI-4 scores PPL 24.121 on the same three books.The passage states that the globally fixed base/fraction has not been shown to beat KIVI-4 across corpora.

5.4 Exploratory low-rate and scaling results

Exploratory results examine low-rate behavior, scaling, and the distinction between lower perplexity and cache accuracy. These results are explicitly limited by evaluation and implementation scope.

  • Scaling: 30B-A3B runs on a 48 GB RTX A6000 demonstrate feasibility for the desired model scale, but the reported Table 4 results require progressive recomputation.The results preceded the evaluation-mode flag and used Q = 64.
  • Low-rate pilot: 7.443 PPL at 1.996 bits/value and 7.335 at 2.993 bits/value were obtained for pure graded points in the 0.8B pilot.KIVI-2 scored 8.905 at 3.033 bits/value, while KIVI-4 scored 7.570 at 5.034 bits/value.
  • Interpretation: Lower perplexity can result from purging far-context information as regularization rather than exact memorization.The paper states that this analysis is not a replacement for retrieval evaluations.

5.6 Packed-kernel microbenchmark

The packed attention-only kernel is faster than the particular dense reference while using a substantially more compact cache, but the benchmark excludes major end-to-end components.

  • Benchmark scope: The microbenchmark reports compact-cache size and packed attention-only speed, not generation throughput or end-to-end peak VRAM.Excluded components include projections, MLPs, sampling, cache append, allocator runtimes, and Transformer dispatches; the quality path retains dense K/V matrices.

5.7 Resident context capacity

Resident context capacity depends on actual allocation, attention architecture, and cache rate rather than rate accounting alone. Compression improves capacity most clearly for concurrency and for contexts exceeding uncompressed device limits.

  • Measurement: Real Q = 1 decoding steps and end-to-end generation benchmarks validate compressed context feasibility, while some compressed limits use buffer allocation rather than real decoding.The Full-KV limit is measured by real decode during binary search; compressed representations are allocated at computed sizes and later verified in vLLM generation.
  • Measurement: 1.959 bits/value measured versus 1.990 predicted shows that Geodesia-KV rate accounting is mildly conservative.
  • Single-sequence capacity: 294,911 tokens fit for uncompressed 8B on an A6000, so native-window capacity is limited by the trained position window rather than cache memory.
  • Single-sequence capacity: At 131,072 tokens, the 8B uses 7.99 GiB with the two-bit branch versus 11.33 GiB with KIVI-4, enabling the supported window on 16 GiB devices.
  • Single-sequence capacity: 2.44× lower rate gives Geodesia-KV 2-bit an advantage over KIVI-4 for both 8B and 3B, while attention architecture determines absolute capacity.The fourfold difference in bytes per token from eight versus two KV heads makes 3B contexts larger under every policy.
  • Concurrency: 8.8× as many concurrent sequences as Full-KV and 2.5× as many as KIVI-4 fit at the native window with the two-bit branch.Quest provides no capacity benefit because its exact resident K/V and page summary cost slightly more than uncompressed storage.

5.8 Lessons Learned

The experiments favor simple scalar precision ladders and uniform allocation over several more elaborate hand-designed alternatives. Multiple proposed corrections, subspaces, profiles, and exact-tail hybrids failed to transfer reliably across holdout windows.

  • Evaluation discipline: Final comparisons must use Q=1 because larger chunks increase KIVI’s current exact residual and alter both rates and perplexity.
  • Transferability: 2.061-bits/value exact-tail hybrid was close to SnapKV during development but underperformed on holdout.
  • Representation choices: Ranks one through four of the covariance-based block state had worse PPL than the scalar ladder despite faster batched power iteration.
  • Transferability: A global second-order correction helped one window but hurt another, so its benefit did not transfer consistently.
  • Transferability: Smallest-subspace selection helped validation windows but failed the frozen test, while combining Born projection with RD-V2 was slower and worse.
  • Representation choices: At practical rates, more scalar bits with coarser groups beat two-bit payloads with expensive fine-grained scaling because similar keys did not imply similar values.
  • Allocation: Uniform allocation was better or comparable to the tested hand-designed six-group layer profiles under equal mean capacity.

6 Limitations & Systems Resolution

A native vLLM cache manager connects Geodesia-KV’s rate-distortion design to physical serving hardware. The systems evaluation reports large VRAM reductions, long-context retrieval, and statistically validated PG-19 comparisons.

  • Systems integration: The GeodesiaKVCacheManager implements 64-token block paging, GQA head sharing, monotone bit demotion, and incremental appending in vLLM 0.26.0.
  • Hardware evaluation: 71.7% VRAM reduction shrinks the FP16 KV cache from 12.25 GiB to 3.47 GiB at 32,768 context and 439.5 tokens/sec on an RTX 4090.
  • Retrieval: 100.0% NIAH retrieval accuracy holds across context depths and multi-turn dialogues, while Geodesia-KV reaches 2.91 BPV and 81.8% VRAM reduction at 16k.
  • Hardware evaluation: 1M-token inference at 2.0 BPV reduces FP16 KV cache from 53.41 GiB to 8.45 GiB, reaching a single 16 GiB consumer GPU.
  • Statistical validation: 1.00516 compressed-Quest perplexity ratio with a 95% interval of [1.00076, 1.00945] avoids 1.0 across the full 11-book PG-19 dataset.

7 Conclusion

Geodesia-KV unifies block-paged KV-cache compression under rate–distortion constraints and combines high retrieval accuracy with substantial VRAM savings. Its vLLM integration enables substantially larger context capacity on consumer hardware.

  • Geodesia-KV manages block-paged KV caches with a unified rate–distortion approach.
  • 2.91 BPV resident capacity corresponds to 81.8% VRAM saving in the real LLM serving profile.
  • Eviction strategies such as StreamingLLM and SnapKV fail multi-turn topic shifts because evicted context cannot be recovered, while Quest adds VRAM resident overheads.
  • 84.2% VRAM savings and 100% retrieval accuracy are achieved with a bit-level ladder spanning 16, 8, 4, 2, and centroid states.
  • The vLLM plugin scales context capacity by over 8.8× and enables 1M-token context inference on a single 16 GiB GPU.
Loading 2608.23599v1…