PaperlayerView exact revision on arXiv

Source-linked AI summary

CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM

Yubo Li, Yidi Miao

arXiv:2605.24786v1cs.LGcs.AI

TL;DR

Long-horizon inference makes KV-cache growth a central memory and latency challenge. CONF-KV uses current uncertainty to adapt cache budgets and, across models and tasks, remains near sliding-window memory while preserving substantially more quality and retrieval.

  • Problem

    KV-cache growth with sequence length can dominate GPU memory and attention latency in long-horizon LLM applications.

  • Method

    CONF-KV converts the next-token distribution into confidence, adapts per-step cache budgets, and ranks retained tokens using attention mass and recency with a protected recent window.

  • Results

    91.4% retrieval accuracy versus 53.8% for sliding windows and 80.6% for H2O, while CONF-KV variants remain near the memory footprint of a 512-token sliding window across four models.

  • Takeaways & Limitations

    Current uncertainty improves the KV-cache memory–quality trade-off over static and historical-attention policies at matched memory.

  • Takeaways & Limitations

    The confidence signal becomes less informative under high-temperature sampling, where entropy saturates and the threshold may require retuning.

Abstract

from arXiv · show

Long-horizon LLM inference turns the key--value (KV) cache into the dominant GPU memory consumer and makes per-token attention increasingly expensive. Many common eviction policies use static recency windows or historical attention, leaving unused a signal computed on every decoding step: the model's current uncertainty. We introduce CONF-KV, a KV-cache manager that converts the next-token distribution into a scalar confidence score and uses it to choose the per-step cache budget, retaining more context when the model is uncertain and pruning aggressively when it is confident. Within each budget, tokens are ranked by a composite of accumulated attention mass and recency, while a protected recent window preserves local coherence. We combine the policy with blockwise online-softmax attention, mixed FP16/INT8 storage, and a pyramidal per-layer budget variant. Across four model families and generated lengths up to 4K, CONF-KV stays near the footprint of a fixed 512-token sliding window while remaining within 1.5--2.1 perplexity points of full KV. On Needle-in-a-Haystack up to 32K tokens, CONF-KV reaches 91.4% retrieval accuracy versus 53.8% for sliding windows and 80.6% for H2O; on 75 VisualWebArena tasks it retains 95.3% of full-KV success at 2.8 times lower peak memory.

1 Introduction

Long-horizon applications make KV-cache growth a major memory and latency bottleneck, while existing policies primarily rely on past-token signals. CONF-KV instead uses the current next-token distribution to adapt cache budgets and combines confidence-aware eviction with systems optimizations and matched-memory evaluations.

  • Motivation: 15.8 GB is the measured full-KV allocator footprint for Qwen-32B during the 4K generated-token sweep, illustrating the cache’s memory burden.The KV cache grows linearly with sequence length and depth and can also dominate attention latency.
  • Motivation: Existing policies use recency, historical importance, prompt observations, or layer variation, leaving the current output distribution underused for cache management.The cited policies include sliding-window attention, H2O, SCISSORHANDS, SNAPKV, and PYRAMIDKV.
  • CONF-KV: CONF-KV maps entropy, log-probability margin, and top-token mass into confidence, selecting a tight or loose budget before ranking evictions by attention mass and recency.A hard protected window preserves the newest tokens; the method requires no training and does not change model weights.
  • Results: 74% is the perplexity-gap closure achieved by CONF-KV-L versus a 512-token sliding window and full KV at matched memory on GPT-2.CONF-KV+INT8 closes 60% of that gap; CONF-KV-L is the pyramidal layer-budget variant using the same FP16/INT8 storage as CONF-KV+INT8.
  • System design: CONF-KV combines adaptive eviction with blockwise attention, cache compaction, mixed FP16/INT8 storage, and an optional pyramidal per-layer budget.The paper also tests whether confidence anticorrelates with the KL shift caused by ablating recent context.

2 Related work

Prior KV-cache work uses fixed windows, attention importance, prompt or head-wise structure, and layer-specific budgets, while efficient-attention and quantization methods address complementary memory or compute dimensions. CONF-KV’s narrow novelty is using next-token confidence to set a per-step token-retention budget during autoregressive decoding.

  • KV-cache eviction: Sliding windows and attention sinks provide fixed-memory streaming, whereas H2O, SCISSORHANDS, SNAPKV, FASTGEN, and PYRAMIDKV use attention, prompt, head-wise, or layer-depth signals.These methods differ in how they identify useful tokens or allocate cache budgets.
  • Adaptive and uncertainty-aware KV compression: Adaptive compression methods allocate budgets using loss guidance, layer uncertainty, or matrix-entropy uncertainty rather than one global cap.Examples include ADA-KV, ZIGZAGKV, and UNCOMP.
  • Efficient attention and serving memory: Efficient-attention and serving methods reduce activation memory, improve KV layout, sparsify reads, or lower precision without deciding which tokens remain cached.FLASHATTENTION, PAGEDATTENTION, QUEST, SPARQ, KIVI, and KVQUANT address dimensions complementary to token retention.
  • Confidence signals: CONF-KV uses next-token confidence to control memory state through a per-step KV token-retention budget during autoregressive decoding.The novelty claim is narrow: uncertainty is not claimed to be new for compression, but this confidence signal is used to adapt retention at each decoding step.

3 Method

CONF-KV uses next-token confidence to select a larger cache budget for uncertain steps and a tighter budget for confident steps. It ranks non-protected tokens by attention mass and recency, while combining compacted storage, mixed precision, and optional layerwise budgets.

  • Confidence estimator: The confidence score combines normalized entropy, log-probability margin, and top-token probability with weights 0.4, 0.3, and 0.3.The score need not be calibrated as a probability; the policy requires only a monotone relation between confidence and context demand.
  • Adaptive cache budget: Confidence selects Nhigh when c ≥ τ and Nlow when c < τ, with Nhigh ≤ Nlow.Nhigh is the tighter budget for confident steps, whereas Nlow is the larger budget for uncertain steps.
  • Token selection: Tokens are ranked by si = αâi + (1 − α)r̂i, using EMA attention mass and recency, while the most recent protected tokens are retained.Attention mass is updated with λ = 0.9, and both signals are min–max normalized over non-protected candidates before interpolation.
  • Tiled attention and mixed precision: Older retained entries are symmetrically quantized to INT8, while the most recent W retained tokens remain in FP16 during blockwise online-softmax attention.Dequantization is fused into blockwise attention reads, and the cache is compacted before serving attention.
  • Pyramidal layer budgets: CONF-KV-L allocates selected budgets non-uniformly across layers, using β = 0.5 and Nmin = 96 unless noted.The layerwise mechanism adapts along depth, while the confidence rule adapts over time.

4 Experimental setup

The experiments evaluate CONF-KV across four model families and three workloads, comparing it with full KV and established cache policies under matched-memory conditions. Results use deterministic WikiText-2 evaluation, three-seed averages for NIAH, VWA, and throughput, and H100-based infrastructure.

  • Models and workloads: Four models—GPT-2 (124M), Qwen-14B, gpt-oss-20b, and Qwen-32B—are evaluated on WikiText-2, NIAH, and VisualWebArena workloads.WikiText-2 uses 512–4096 generated tokens; NIAH spans 1K–32K haystacks and five needle depths; VWA serializes rendered pages into visible text without raw image tokens.
  • Baselines and configurations: Baselines include full KV, fixed 512-token sliding windows, H2O, SCISSORHANDS, SNAPKV, and PYRAMIDKV, tuned to match CONF-KV+INT8 average peak KV memory.All methods use the same tiled attention path to isolate cache-policy effects; default thresholds and parameters vary by workload.
  • Statistics and compute: WikiText-2 uses deterministic greedy decoding and is reported once, whereas NIAH, VWA, and throughput report mean ± standard deviation over three seeds.Experiments run on NVIDIA H100 80 GB GPUs with CUDA 12.8, PyTorch 2.9, and Transformers 4.51.

5 Results

CONF-KV improves matched-memory quality and preserves long-context retrieval by adapting cache budgets to confidence, while mixed-precision storage keeps memory near a 512-token sliding window. It also reduces latency and retains strong task performance on VisualWebArena.

  • Matched-memory quality: 3.89 PPL points: CONF-KV-L improves over the sliding window at matched memory and closes 74% of the full-cache/sliding-window quality gap.CONF-KV+INT8 improves perplexity by 3.11 points; CONF-KV-L uses less memory than every baseline.
  • Matched-memory quality: 63% of the full-cache/sliding-window gap is closed by PYRAMIDKV, versus 74% for CONF-KV-L.Static caps must over-provision routine steps or under-provision rare difficult ones.
  • Memory scaling: 13.2 GB: CONF-KV reduces Qwen-32B KV memory from 15.8 GB to 2.6 GB at 4K generated tokens with matched prefill state.Across four models, peak memory remains in the same range as a 512-token sliding window, reaching about 1.3× its footprint at the longest sweep.
  • Confidence signal: 30.92 PPL: full CONF-KV outperforms attention-only, recency-only, random matched-rate eviction, and the fixed sliding window.The matched-rate baselines use the same per-step eviction probability and number of evicted tokens per event.
  • Long-context retrieval: 91.4% retrieval accuracy: CONF-KV outperforms sliding windows and H2O on long-context Needle-in-a-Haystack retrieval by expanding its budget when confidence drops.Raising τ from 0.7 to 0.8 recovers most remaining failures at a 12% memory cost.
  • VisualWebArena: 95.3% of full-KV task success: CONF-KV achieves this on 75 VisualWebArena tasks while reducing peak memory by 2.8×.Full-KV success is 40.2%, CONF-KV reaches 38.3%, and sliding-window truncation loses 11.1 absolute points.
  • Latency and profiling: 1.8× lower p50 latency: CONF-KV achieves this on GPT-2 and Qwen-32B relative to full KV at 2048 generated tokens.Attention falls from 62% of full-KV step time to 47%; compaction adds 0.22 ms and metadata updates add 0.11 ms on GPT-2.

6 Ablations and sensitivity

The ablations identify stable operating points for confidence thresholds, high-confidence budgets, and FP16 windows, while showing that CONF-KV’s cache size dynamically follows model confidence rather than remaining fixed. Precision and layer-allocation sweeps quantify the quality–memory trade-offs of storage choices.

  • Threshold and budget: τ = 0.7 is a stable operating point, while lower thresholds reduce memory at a quality cost and higher thresholds improve quality at a memory cost.The confidence threshold controls how often the policy selects the tight budget.
  • Threshold and budget: Nhigh = 128 tokens is the WikiText-2 knee: smaller budgets make confident-step evictions too aggressive, while larger budgets increase memory faster.The Nhigh sweep identifies a similar knee at 128 tokens.
  • Precision and layer allocation: W = 128 is the best GPT-2 operating point; smaller windows expose recent tokens to quantization error, whereas larger windows reduce memory savings.The FP16 window controls how much recent cache avoids quantization.
  • Precision and layer allocation: 0.38% mean roundtrip error and 0.34 PPL relative to FP16 are incurred by INT8 per-(head,channel) scaling, while NF4 and INT4 add 0.91 and 1.65 PPL beyond INT8, respectively.These results quantify the precision trade-off for mixed-storage cache designs.
  • Temporal behavior: The cache follows a sawtooth trajectory, growing during uncertain or below-threshold steps and compacting when confidence rises and a tight budget is selected.The confidence histogram is bimodal enough to provide frequent high-confidence compaction opportunities while preserving a heavy tail of low-confidence steps.

7 Implementation notes and failure modes

CONF-KV’s contiguous compaction keeps the attention kernel unchanged but adds modest eviction-time overhead. Its main failure mode is confidently wrong pruning of rare, earlier context, while deployment can use alternative layouts with coarsened or sparse eviction.

  • Implementation notes: 0.22 ms per step for compaction and 0.11 ms for metadata updates remain smaller than the attention savings from the reduced cache.The GPT-2 profile measured these costs at the observed eviction rate, with bounded post-compaction live memory.
  • Failure modes: Remaining NIAH and VWA failures arise when the model is confidently wrong and evicts rare entities, product names, or form fields needed later.Raising τ recovers NIAH examples at a measurable memory cost, indicating a tunable trade-off.
  • Implementation notes: CONF-KV is layout-agnostic, but token-level eviction in PAGEDATTENTION-style layouts can create partially dead blocks unless eviction is coarsened or sparse masks are added.The method is also orthogonal to speculative decoding, allowing a draft model to propose candidates while the target model accepts steps.

8 Limitations and conclusion

CONF-KV has limited benefit on short contexts, MLP-bound workloads, and high-temperature sampling, while fine-grained eviction complicates paged serving. Lower KV memory can reduce inference cost and energy but may also reduce the cost of undesirable automation, so deployments should retain underlying safety controls.

  • Limitations: CONF-KV is inactive on short contexts below the eviction threshold and offers smaller speedups when MLP computation dominates latency.Its benefits are larger when attention or KV bandwidth dominates.
  • Limitations: High-temperature sampling makes confidence less informative because entropy saturates, potentially requiring τ retuning.
  • Limitations: Fine-grained token eviction complicates mapping the simple contiguous compaction strategy onto paged serving layouts.Contiguous compaction is simple and fast enough in the reported setting.
  • Safety and conclusion: Lower KV-cache memory can reduce inference cost and energy use but may also lower the cost of undesirable long-horizon automation.
  • Safety and conclusion: CONF-KV adds no new model capabilities, so deployments should inherit the safety controls used by the underlying agent or LLM.The approach remains compatible with paging, quantization, and speculative decoding.

A Extended quantitative results

Table 3 reports measured peak KV-memory allocator footprints across models and generated-token counts. The continuation sweep preserves a matched prefill/prefix cache state, and measurements include temporary cache-management workspaces rather than only live post-compaction tokens.

  • Peak KV-memory footprint: Table 3 measures peak KV-memory allocator footprint in MB across all models and generated-token counts.The reported quantity is allocator footprint, not simply the number of retained cache tokens.
  • Measurement protocol: The continuation sweep keeps a matched prefill/prefix state in the cache during measurement.This condition controls the cached state used for the continuation comparison.
  • Measurement protocol: Measurements include temporary cache-management workspaces, so they are not identical to live post-compaction token counts.Peak allocator usage therefore reflects transient management memory as well as the cache footprint.

B Additional ablations and traces

Additional ablations show that INT8 offers the best quantization trade-off, while adaptive compaction keeps live KV memory bounded during decoding. The traces support memory savings without unbounded growth.

  • Quantization analysis: INT8 provides most cache-memory reduction with lower roundtrip error than NF4 or INT4.Figure 9 compares the quantization alternatives.
  • Memory trace: Adaptive compaction produces bounded sawtooth behavior in live KV memory over 1200 generation steps.The traced decode exhibits repeated bounded memory variation rather than unbounded growth.

C Confidence calibration across models

Table 4 reports Pearson correlations between the confidence score and the KL shift caused by ablating the past 256 tokens, evaluating confidence calibration across models.

  • Table 4 measures the Pearson correlation between confidence score and KL shift from ablating the past 256 tokens.

D Reproducibility details · NeurIPS Paper Checklist

The paper documents a modular, single-GPU experimental setup and reports reproducibility, limitations, statistical reporting, asset licensing, and release decisions through the NeurIPS checklist. It identifies absent runnable code or data at submission while describing planned release and the training-free evaluation setting.

  • D Reproducibility details: Experiments run on a single NVIDIA H100 80 GB GPU unless otherwise noted, with VisualWebArena hosted on official AWS EC2 infrastructure.The implementation separates the confidence-aware generator, cache manager, quantizer, and tiled attention backend.
  • D Reproducibility details: Appendix D records hyperparameter settings, existing assets, licenses, terms status, and approximate reproduction compute budgets.The compute-budget table notes dependence on checkpoint, sequence length, serving environment, and EC2-hosted sites for VisualWebArena.
  • NeurIPS Paper Checklist: The paper states that its abstract and introduction accurately present the method, scope, baselines, and empirical claims reported in the Results section.The checklist answer is supported by the correspondence between stated claims and reported measurements.
  • NeurIPS Paper Checklist: Section 8 discusses limitations involving short contexts, high-temperature sampling, compute bottlenecks, and compaction overhead.These limitations address operational conditions and costs relevant to the method’s use.
  • NeurIPS Paper Checklist: The paper marks theory assumptions and proofs as N/A because it is an empirical systems paper without theoretical theorems or proofs.The checklist therefore does not require a complete proof treatment for this work.
  • NeurIPS Paper Checklist: The submission does not include anonymized runnable code or data, while a first-page footnote states that code will be released upon acceptance.This is the stated open-access limitation for reproducing the main experimental results at submission time.
  • NeurIPS Paper Checklist: The experimental details specify models, benchmarks, baselines, decoding settings, hyperparameters, seeds, and hardware, with no optimizer because the method is training-free.The checklist indicates that Section 4 and Appendix D provide these details.
  • NeurIPS Paper Checklist: NIAH, VWA, and throughput are reported as mean ± standard deviation over three seeds, while mechanistic validation reports a correlation and p-value.The checklist also emphasizes defining variability factors, error-bar calculations, assumptions, and whether error bars represent standard deviation.
Loading 2605.24786v1…