PaperlayerView exact revision on arXiv

Source-linked AI summary

SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

Amirhossein Abaskohi, Giuseppe Carenini, Peter West, Yuhang He

arXiv:2606.31145v1cs.CL

TL;DR

Long-context LLM inference must preserve query-relevant evidence under tight memory constraints, but existing compression methods may lose token-level detail when compressed spans later become relevant. SeKV uses resolution-adaptive semantic memory to dynamically reconstruct relevant spans, improving over SentenceKV by 5.9% on average while reducing GPU memory by 53.3% versus full KV caching at 128K context.

  • Problem

    Long-context LLM inference must preserve query-relevant evidence under tight memory constraints, but existing compression cannot recover detail from spans later deemed relevant.

  • Method

    SeKV organizes context into entropy-guided semantic spans with GPU summaries and CPU low-rank bases, then dynamically zooms into query-relevant spans for token-level reconstruction.

  • Results

    5.9% average improvement over SentenceKV across four benchmarks, while reducing GPU memory 53.3% versus full KV caching at 128K context.

  • Takeaways & Limitations

    The evaluated results support query-adaptive cache resolution as an effective alternative to static KV compression for long-context inference.

  • Takeaways & Limitations

    SEKV’s effectiveness depends on semantic span quality; noisy boundaries or poorly calibrated uncertainty estimates can cause suboptimal memory allocation.

Abstract

from arXiv · show

Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be retained throughout decoding, making full GPU caching prohibitively expensive without compression. Existing KV cache compression methods struggle to balance efficiency with faithful context preservation. Token eviction discards information, while semantic grouping fixes compression decisions at prefill time; neither can recover token-level detail from a compressed span once it becomes relevant during generation. As a solution, we propose SeKV, a resolution-adaptive semantic KV cache that organizes context into entropy-guided semantic spans and stores them across a GPU-CPU memory hierarchy without discarding information. Each span keeps a lightweight summary vector on GPU for coarse routing and a low-rank SVD basis on CPU for on-demand token-level reconstruction. A trained zoom-in mechanism selectively expands query-relevant spans during decoding, enabling precise retrieval without materializing the full KV cache on GPU. SeKV enables adaptive token-level reconstruction while keeping the base LLM fully frozen and adding fewer than 0.05% trainable parameters. Across four benchmarks, SeKV improves over the strongest semantic compression baseline by 5.9% on average while reducing GPU memory by 53.3% versus full KV caching at 128K context. Code is available on https://github.com/AmirAbaskohi/SeKV.

1 Introduction

Long-context inference is difficult because KV-cache memory grows linearly with sequence length, while existing compression methods may discard information or fix semantic compression before decoding. SeKV addresses this with hierarchical, resolution-adaptive storage that preserves information and reconstructs token-level detail for query-relevant spans on demand.

  • Limitations of prior methods: Existing methods either compress token-level states or apply static semantic grouping, which can fragment evidence, discard information, or fail to recover detail when relevance changes during decoding.This is especially problematic when initially peripheral evidence becomes a later query’s logical anchor.
  • SeKV: SeKV organizes context into hierarchical semantic spans and dynamically adjusts cache resolution according to query relevance during decoding.Its entropy-guided segmentation uses token surprisal as a boundary signal, including topic shifts and entity introductions.
  • SeKV: GPU-resident summary vectors provide coarse routing, while CPU-resident low-rank bases enable approximate token-level reconstruction without irreversible token eviction.A trained zoom-in mechanism selectively expands query-relevant spans during decoding, while high-surprisal anchor tokens remain at full GPU resolution.
  • Results: 5.9% improvement over the strongest semantic compression baseline is achieved across four long-context understanding benchmarks, while GPU memory consumption is reduced by 53.3% compared to full KV caching at 128K context length.These results combine adaptive retrieval with substantially lower GPU memory use.

2 Related Work

Prior KV-cache compression largely relies on permanently evicting tokens or using semantic units and cache representations fixed during prefill, limiting later fine-grained recovery. SeKV addresses this with entropy-guided spans, hierarchical summaries and bases, and on-demand query-relevant reconstruction.

  • KV cache compression and eviction: Token eviction permanently discards tokens according to attention statistics and dominates KV-cache compression.The passage cites Zhang et al. (2023), Liu et al. (2023), Xiao et al. (2024b), Li et al. (2024), and Cai et al. (2025).
  • KV cache compression and eviction: Fixed semantic units and retained cache representations make it difficult to recover finer-grained evidence outside the retained token pool when it becomes relevant later.
  • KV cache compression and eviction: SeKV uses entropy-guided spans, GPU-resident routing summaries, CPU-resident low-rank bases, and trained zoom-in to reconstruct query-relevant spans on demand.

3 SEKV

SeKV builds a hierarchical semantic KV memory by segmenting context with surprisal-guided boundaries and representing each span at GPU summary and CPU low-rank resolutions. During decoding, it routes over summaries, asynchronously reconstructs selected spans, and combines coarse and token-level attention without dropping context.

  • Dual-resolution representation: Each span stores a compact summary vector on GPU for coarse routing and a low-rank SVD basis on CPU for on-demand token-level reconstruction.The summary routing dimension is d′ = 32, while SVD components are gated per span using a learned rank budget.
  • Semantic segmentation: SeKV uses token-level surprisal to create content-adaptive semantic spans, retaining high-surprisal boundary tokens as full-resolution GPU anchors.Boundaries occur when H_t > µ + ασ; surprisal is already produced during prefilling, requiring no additional computation.
  • Adaptive zoom-in: Expanded spans fetch CPU SVD bases asynchronously, reconstruct token-level KV pairs, and enter the same softmax attention as coarse summaries, so no span is dropped.The mixed-resolution computation combines persistent anchors, coarse span entries, and reconstructed tokens in a single attention operation.
  • Adaptive zoom-in: At each decoding step, per-head, per-layer routing probabilities and thresholds select spans for expansion, while non-expanded spans contribute coarse mean key-value entries.Per-head thresholds accommodate different head selectivity, including retrieval heads that expand many spans and streaming heads that rarely expand.
  • Memory budgeting: The peak GPU-resident KV budget includes anchors, summaries, coarse means, and reconstructed spans; thresholds set expansion, while a relevance-ranked cap enforces the matched budget.When selected spans exceed the budget, spans are retained in decreasing order of relevance probability until capacity is filled.

4 Experiments and Results

Across four long-context benchmarks and five backbones, SEKV is the strongest compressed-cache method, improving over SentenceKV while retaining retrieval and reasoning quality. It also reduces runtime and GPU-memory growth through selective, head- and layer-specific span reconstruction.

  • Long-context benchmark performance: SEKV wins all 20 benchmark–model comparisons and improves over SentenceKV by 5.9% on average under the same 10% KV budget.The evaluation covers LongBench, RULER, InfiniteBench, and NIAH across five backbone models.
  • Needle retrieval behavior: SEKV maintains stable NIAH retrieval across context lengths and needle depths, while SentenceKV, ChunkKV, and token-eviction methods show broader low-success regions.The result indicates that SEKV’s gains extend across the length–depth space rather than concentrating on easy needle positions.
  • Many-shot reasoning: SEKV achieves the best compressed-cache GSM8K result across all five backbones, improving over SentenceKV by +2.3 points on average and remaining within 2.4 points of FullKV.GSM8K tests preservation of many-shot reasoning structure rather than retrieval of a single hidden fact.
  • Runtime and memory: SEKV reduces FullKV latency from 43.60s to 38.05s at 4K/1K and from 183.42s to 166.95s at 8K/4K, while increasing throughput from 105.92 to 120.11 T/S and from 55.93 to 64.21 T/S.Among semantic methods, SEKV offers the best trade-off between speed and accuracy.
  • Runtime and memory: GPU memory rises only from 31.2GB to 34.9GB between 8K and 128K for SEKV, versus FullKV’s increase from 36.0GB to 74.8GB.Zoom-in decisions are sparse and concentrated in specific heads, especially in mid-to-late layers, rather than applied uniformly.

5 Conclusion

SeKV is a resolution-adaptive semantic KV cache for efficient long-context LLM inference. It uses entropy-guided spans, GPU routing summaries, and on-demand reconstruction of low-rank CPU-resident span bases without modifying the base LLM.

  • 5 Conclusion: SeKV introduces a resolution-adaptive semantic KV cache for efficient long-context LLM inference.The method is designed for long-context inference.
  • 5 Conclusion: Entropy-guided spans organize context while lightweight routing summaries remain on GPU.This organization supports query-adaptive cache resolution.
  • 5 Conclusion: Low-rank CPU-resident span bases are reconstructed on demand, enabling query-adaptive cache resolution without modifying the base LLM.Reconstruction occurs selectively during inference rather than requiring full materialization in the GPU cache.
  • 5 Conclusion: SeKV outperforms prior KV compression methods, including token-eviction approaches such as StreamingLLM, H2O, SnapKV, and PyramidKV.The cited prior methods are identified as token-eviction approaches in the conclusion passage.

Limitations

SeKV’s effectiveness depends on reliable entropy-guided semantic span construction, while freezing the base LLM and using lightweight routing may limit adaptation to the compressed memory structure. Future work includes jointly training routing with richer span representations and adaptive reconstruction policies.

  • SeKV’s effectiveness depends on the quality of entropy-guided semantic span construction.Noisy boundaries or poorly calibrated uncertainty estimates can produce suboptimal outcomes.
  • Freezing the base LLM and adding only lightweight routing components may limit adaptation to the compressed memory structure.
  • Jointly training routing with more expressive span representations or adaptive reconstruction policies is identified as future work.

A Training Details

SeKV trains lightweight routing components while keeping the backbone frozen, using a single disjoint RedPajama long-document corpus. Training distills teacher attention and reconstruction behavior, supervises zooming at mined non-local query positions, and uses a 32K-length curriculum with AdamW optimization.

  • Training data: SeKV trains only lightweight routing components on long documents from RedPajama, using a single language-modeling corpus while keeping the backbone frozen.The corpus includes arXiv, books, and code subsets.
  • Training data: Zoom supervision uses mined query positions where teacher attention concentrates on non-local spans, whose attended spans define the coverage target Cρ.Distillation and reconstruction objectives use all positions.
  • Training data: The training corpus is disjoint from every Section 4.1 benchmark and contains no synthetic retrieval or instruction data; routing parameters are trained separately for each backbone.This aligns routing with the teacher’s long-range attention without QA-template distribution shift.
  • Optimization: AdamW training uses a 1 × 10−3 learning rate, 0.01 weight decay, cosine decay, 10% warmup, roughly 3K steps, and about 0.5B tokens.Training uses a maximum sequence length of 32K, an 8K →32K curriculum, held-out early stopping, and budget-weight annealing.

B Benchmark and Baseline Details · B.1 Benchmark Details · B.2 Baseline Details

The evaluation spans diverse long-context benchmarks measuring understanding, controlled context use, retrieval fidelity, ultra-long reasoning, and many-shot reasoning. Baselines cover uncompressed caching, token and chunk retention, structured allocation, and semantic compression methods with varying adaptability and recoverability.

  • B.1 Benchmark Details: SEKV is evaluated on LongBench, RULER, InfiniteBench, NIAH, and 50-shot GSM8K, covering understanding, retrieval, reasoning, and long-context in-context learning.The benchmarks stress multi-task document understanding, controlled retrieval, ultra-long-context reasoning, needle retrieval, and many-shot mathematical reasoning.
  • B.1 Benchmark Details: LongBench contains 17 datasets spanning question answering, summarization, few-shot learning, synthetic retrieval, and code completion, with task-dependent inputs of roughly 1K–18K tokens.Scores use official task-specific averages with F1, exact match, accuracy, or ROUGE-L, depending on the task.
  • B.1 Benchmark Details: RULER tests controlled context utilization from commonly 4K to 128K tokens using synthetic retrieval and reasoning tasks that place information deep inside contexts.Unlike LongBench’s naturally varying documents and formats, RULER directly varies sequence length.
  • B.1 Benchmark Details: InfiniteBench often exceeds 100K-token inputs and evaluates long-document question answering, summarization, code debugging, and key-value retrieval under ultra-long contexts.The benchmark is important because FullKV can become expensive or infeasible on a single GPU; results follow official task-level aggregation.
  • B.1 Benchmark Details: NIAH averages retrieval accuracy across context lengths and needle depths, while 50-shot GSM8K reports exact-match accuracy after answer normalization.NIAH retrieves an inserted fact from distractors; GSM8K tests retaining many demonstrations that define an in-context reasoning pattern.
  • B.2 Baseline Details: The baselines span token eviction, importance-based retention, structured chunk compression, semantic cache compression, and FullKV as an uncompressed reference.FullKV retains every key and value state, maximizing fidelity but incurring memory that grows linearly with context length.
  • B.2 Baseline Details: StreamingLLM, H2O, SnapKV, PyramidKV, and ChunkKV retain or evict token-level states, but their discarded evidence cannot be recovered or their decisions cannot adapt during decoding.StreamingLLM keeps sink and recent-window tokens; H2O uses accumulated attention importance; SnapKV uses prefill attention; PyramidKV allocates budgets by layer; ChunkKV selects contiguous chunks.
  • B.2 Baseline Details: SemantiCache merges semantically grouped tokens into fixed representative states, while SentenceKV combines sentence representations with GPU–CPU management but relies on static sentence segmentation.SentenceKV is described as the closest baseline because it retrieves relevant token-level states during decoding; SEKV instead uses entropy-guided spans and low-rank span bases.

C Ablation Studies

Ablations show that token-level reconstruction, trained zoom-in, and entropy-guided segmentation are central to SEKV’s performance, while budget and rank-cap sensitivity reveal strong effectiveness under constrained memory and saturation beyond moderate settings.

  • Component ablations: Removing SVD reconstruction or trained zoom-in reduces NIAH accuracy from 91.17 to 83.47 and 85.96, respectively.Span summaries alone are insufficient because SEKV must recover token-level detail when a span becomes relevant.
  • Component ablations: Fixed-chunk segmentation substantially hurts performance, especially on RULER and NIAH, compared with entropy-guided segmentation.This supports using entropy-guided semantic spans rather than fixing compression boundaries in advance.
  • KV-budget sensitivity: At 5% KV budget, SEKV reaches 86.31 NIAH accuracy, outperforming SentenceKV by +7.69 points and ChunkKV by +12.00 points.SEKV consistently outperforms token-level and semantic compression baselines across GPU-resident KV budgets.
  • KV-budget sensitivity: Increasing the KV budget from 5% to 10% improves SEKV from 86.31 to 91.17 NIAH accuracy, while the 10%-to-20% increase adds only +1.29.Most useful long-context evidence is recovered with a small GPU-resident cache, after which gains saturate.
  • Hyperparameter sensitivity: Raising the maximum rank cap R from 8 to 32 improves performance across all four benchmarks, while increasing it to 64 yields only marginal changes.A tight cap can clip spans requiring more reconstruction components; gains saturate beyond R = 32.

D Memory and Complexity Analysis

SEKV replaces FullKV’s linear, full-resolution GPU cache with a variable-resolution GPU–CPU hierarchy that routes using compact span summaries and reconstructs selected token-level states on demand. This reduces GPU growth with context length while retaining recoverable low-rank structure, though reconstruction is approximate rather than exactly lossless.

  • FullKV memory cost: 24GB of KV-cache memory is required by FullKV alone at 128K tokens for the example 48-layer model with bf16 KV states.The cache requires approximately 192KB per token, excluding weights, activations, buffers, and runtime overhead.
  • SEKV GPU memory cost: SEKV stores anchor and local-window KV states plus compact span summaries on GPU, while moving recoverable low-rank span bases to CPU.Its GPU memory therefore grows primarily with anchors, active local tokens, and spans rather than every context token.
  • SEKV CPU storage cost: SEKV avoids permanent token eviction but provides approximate reconstruction because truncated SVD retains only the top r singular directions.Relevant spans are reconstructed on demand from preserved low-rank token-level structure.
  • Transfer and reconstruction complexity: SEKV transfers and reconstructs only zoomed spans, so decoding overhead depends on sparse zoom-in selections rather than the full context length.This is substantially smaller than full token-level transfers when r ≪dh.
  • Routing and reconstruction complexity: Routing compares queries with m span summaries instead of all n tokens, making the coarse stage cheaper when m ≪n and d′ ≪dh.Token-level reconstruction is incurred only for selected spans.

E Additional Zoom-In Behavior Analysis

SEKV’s zoom-in behavior is sparse and specialized across three backbone models and three long-context benchmarks, with activity concentrated in selected heads and generally stronger in middle and later layers. Benchmark-specific patterns reflect task demands, while confirming adaptive allocation of token-level resolution.

  • Visualization setup: Figure 6 visualizes average zoom-in rates across layers and attention heads for SEKV on NIAH, RULER, and InfiniteBench with three backbone models.Brighter heatmap regions indicate heads that more frequently trigger token-level reconstruction.
  • Overall finding: The visualizations confirm that SEKV allocates token-level resolution adaptively across tasks and model configurations.This conclusion follows from the differing heatmap patterns across benchmarks, layers, and attention heads.
  • Cross-benchmark trends: Zoom-in is sparse, with most attention heads remaining inactive across most layers rather than uniformly expanding compressed spans during decoding.This pattern indicates selective, not global, token-level reconstruction.
  • Cross-benchmark trends: Higher zoom-in rates concentrate in a limited number of attention heads, suggesting that only a subset specializes in long-range evidence recovery.Zoom-in activity is generally stronger in middle and later layers, associated with long-range retrieval and semantic integration.
  • Benchmark differences: NIAH shows sparse, sharp patterns for specific fact recovery, RULER broader routing for multiple retrieval and reasoning tasks, and InfiniteBench the most distributed behavior for longer, heterogeneous inputs.These differences align zoom-in allocation with benchmark task characteristics.
Loading 2606.31145v1…