PaperlayerView exact revision on arXiv

Source-linked AI summary

KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference

Srihari Unnikrishnan

arXiv:2608.21362v1cs.AIcs.DC

TL;DR

Existing prefix caching cannot efficiently reuse shared prompt content when it appears away from the leading position. KVBoost enables position-independent chunk-level KV reuse with dual-hash matching and seam repair, achieving a 4.49× mean TTFT speedup over full recomputation without output-quality regression on its evaluation.

  • Problem

    Existing prefix caching requires shared content to form a leading contiguous prefix, leaving arbitrary-position reuse insufficiently supported despite repeated prompt content.

  • Method

    KVBoost combines dual-hash chunk identification with selective or deviation-guided recomputation to reuse KV tensors across positions while repairing attention seam errors.

  • Results

    4.49× mean TTFT speedup over full recomputation and 16% improvement over vLLM prefix caching were achieved, with 99.2% exact-match accuracy versus 99.1% for both baselines.

  • Takeaways & Limitations

    KVBoost extends practical KV-cache reuse to prompts whose shared content is distributed across arbitrary positions while preserving output quality in the evaluated workload.

  • Takeaways & Limitations

    Evaluation covers only bug localization with short outputs, leaving performance on long-form generation, code completion, and multi-document summarization untested.

Abstract

from arXiv · show

Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing prefix-caching systems reduce this cost but require prompts to share a leading contiguous prefix, limiting effectiveness when shared content appears at arbitrary positions. We present KVBoost, a chunk-level KV cache reuse system for HuggingFace-compatible decoder models that enables reuse regardless of content position. KVBoost introduces a dual-hash keying scheme that separates positional identity (prefix hash) from content identity (content hash), supporting both exact and approximate cache matches. To address attention boundary errors from independently cached chunks, KVBoost employs two repair strategies: SelectiveRecompute, which re-encodes boundary regions, and CacheBlendRecompute, which identifies and recomputes high-deviation tokens after a probe pass. The system further incorporates asymmetric KV quantization (int8/int4), adaptive chunk boundary splitting, and importance-weighted eviction under a fixed memory budget. Evaluated on Qwen/Qwen2.5-3B over 1,000 bug-localization samples, KVBoost achieves a 4.49x reduction in time-to-first-token (142.4 ms vs.\ 639.1 ms) and outperforms prefix caching by 16%, with no loss in accuracy (99.2% vs.\ 99.1%). KVBoost provides a practical, memory-bounded inference acceleration layer compatible with RoPE-based models without architectural modification.

1 Introduction

KVBoost targets prefill latency caused by recomputing KV tensors and overcomes prefix caching’s leading-prefix restriction through position-independent chunk-level reuse. It combines dual-hash matching with seam repair and memory-aware cache mechanisms for RoPE-based HuggingFace models.

  • Chunk-level reuse: KVBoost segments prompts into default 128-token chunks and reuses matching cached KV tensors wherever shared content appears, regardless of prompt position.This addresses prefix caching’s requirement for a common leading prefix.
  • Dual-hash keying: Dual-hash chunk keys separate positional identity from content identity, enabling exact prefix-hash reuse and approximate content-hash reuse with mandatory repair.Corrected position_ids support reuse with RoPE-based models despite position-dependent cached representations.
  • Seam repair: Two seam-repair strategies address missing cross-chunk attention: fixed-window SelectiveRecompute and deviation-guided CacheBlendRecompute.SelectiveRecompute re-encodes boundary windows, while CacheBlendRecompute identifies high-deviation tokens after a probe pass.
  • Memory management: KVBoost adds importance-weighted LRU eviction under a hard memory budget, using each chunk’s KV-tensor ℓ2 norm as an importance proxy.The system also includes asymmetric KIVI-style quantization with per-channel key and per-token value quantization.
  • Cache fidelity: The system improves cache fidelity through adaptive linguistic boundary splitting plus overlap and attention-sink token injection during cache population.These mechanisms target boundary-token quality and chunk placement.
  • Implementation: KVBoost provides two-tier hot in-memory and optional memory-mapped disk storage, with an open-source implementation compatible with RoPE-based HuggingFace models.The implementation is available at https://github.com/pythongiant/kvboost.

2 Related Work

KVBoost extends KV-cache reuse beyond contiguous prefix sharing by retrieving chunks across positions and combining this with deviation-guided recomputation. It complements prior cache-management, quantization, compression, and retrieval approaches through chunk-level reuse and shared-document prewarming.

  • vLLM’s PagedAttention and prefix caching operate on pages and require prompts to share a contiguous prefix, whereas KVBoost lifts this constraint at chunk level.KVBoost reuses cached chunks regardless of their content position.
  • SGLang’s RadixAttention performs longest-prefix matching, while KVBoost’s content-hash tier reuses chunks appearing at different positions across requests.RadixAttention uses a radix tree of cached KV blocks but still requires prefix-level sharing.
  • CacheBlend measures KV deviation after assembling pre-cached chunks and recomputes only high-deviation tokens, an insight implemented by KVBoost’s CacheBlendRecompute.KVBoost integrates this recomputation strategy with its broader dual-hash design.
  • KIVI shows that KV caches support low-bit quantization by exploiting distinct key and value outlier distributions; KVBoost applies asymmetric int8 and int4 quantization to cached chunks.Keys have channel-varying outliers, while values have token-position-varying outliers.
  • LLMLingua, SnapKV, and PyramidKV reduce input or cache size through compression or pruning, whereas KVBoost retrieves cached KV tensors and is orthogonal to these methods.The prior methods selectively drop tokens or prune attention heads or layers.
  • RAG naturally creates workloads for chunk-level reuse because retrieved documents are prepended to prompts, and KVBoost’s warm() API pre-populates their KV tensors for later retrieval.Subsequent queries can retrieve the shared documents’ cached KV tensors directly.

3 Background

Decoder-only transformers use KV caches to avoid recomputing prior keys and values during decoding, but prompt prefill remains costly for long inputs. RoPE position dependence and context mismatches create key challenges for arbitrary-position, chunk-level KV reuse.

  • KV Caching: During prefill, all T prompt tokens produce KV tensors of shape [L,2,T,H,d], making prefill the dominant inference cost for long prompts.The cache stores keys and values for prior positions so subsequent decoding steps avoid recomputation.
  • RoPE Positioning: RoPE-baked keys cannot be directly reused at a new position, creating a position collision that requires rotation correction.A key at position t = 50 reused at t = 1050 requires correction R1000θ.
  • Chunk Reuse: Chunk-level reuse can produce KV tensors conditioned on a prior context [C′1,C2] rather than the current context [C1,C2].This mismatch occurs when a cached chunk follows different preceding content in another prompt.
  • Chunk Reuse: Seam error is largest near a reused chunk’s start and diminishes toward its end, making it the primary quality risk of chunk-level reuse.The pattern follows causal attention and attention score decay with distance.

4 KVBoost System Design

KVBoost is a chunk-level KV reuse pipeline that supports position-independent matching while repairing boundary and positional errors before inference. Its memory-bounded implementation combines adaptive chunking, quantization, importance-weighted eviction, and RoPE-compatible model integration.

  • Chunking: ChunkRegistry segments prompts into fixed-size chunks, with FIXED as the default and SEMANTIC or DOCUMENT alternatives.The default chunk size is C = 128; SEMANTIC favors paragraph or sentence boundaries, while DOCUMENT caches an entire input as one chunk.
  • Hashing and lookup: KVBoost separates positional identity from content identity using prefix hashes for exact reuse and content hashes for approximate reuse.Content-hash matches are position-independent but retain original RoPE rotations, so they require mandatory CacheBlendRecompute.
  • Seam repair: SelectiveRecompute re-encodes the last R tokens at each cached-chunk seam with full preceding context.With R = 16, recomputation costs O(R·Nseams) tokens and is typically ∼8% of full prefill for two seams in a 512-token prompt, but may miss mid-chunk deviations.
  • Seam repair: CacheBlendRecompute repairs deviations by probing cached tokens, selecting the highest-deviation positions, and replacing their KV tensors before the main forward pass.Unlike spatial seam repair, it can identify mid-chunk tokens affected by globally important context and is mandatory for content-hash matches.
  • Compatibility: KVBoost supports decoder models with RoPE positional embeddings and a past_key_values interface, while rejecting unsupported positional or attention schemes.The compatibility checker raises a RuntimeError at from_pretrained time for unsupported models.

5 Implementation

KVBoost is implemented as a typed Python package built on PyTorch and HuggingFace Transformers, with metadata-rich chunk assembly and compatibility mechanisms for Transformer-version differences. Its core functionality requires no dependencies beyond PyTorch, Transformers, and Accelerate.

  • Core package: KVBoost is implemented in Python 3.9+ using PyTorch and HuggingFace Transformers.
  • Core package: CachedChunk stores lookup, eviction, and repair metadata, while AssembledPrompt carries merged KV tensors, live tokens, positions, boundaries, and approximate-match state.
  • Version compatibility: KVBoost probes the model’s forward signature once to cache the correct logits parameter across Transformers versions.Transformers ≥4.45 uses logits_to_keep instead of num_logits_to_keep.
  • Version compatibility: The last_logit_only(model) context manager reduces vocabulary projection from [batch,T,V] to [batch,1,V] during long prefills.It temporarily replaces the LM head with a last-position-only projection, particularly benefiting large-vocabulary models such as Qwen2.5-3B.
  • Packaging: Core functionality requires only PyTorch, Transformers, and Accelerate, and the package provides full type annotations with py.typed compliance.

6 Experiments

Experiments on Qwen/Qwen2.5-3B use a 1,000-sample bug-localization benchmark to compare full recomputation, vLLM prefix caching, and KVBoost. KVBoost preserves accuracy while substantially reducing TTFT, with benefits increasing for longer contexts despite slightly lower mean cache reuse.

  • Experimental setup: Experiments use Qwen/Qwen2.5-3B with float16 HuggingFace Transformers inference on a single NVIDIA GeForce RTX 4060 GPU.No tensor parallelism, distributed inference, or additional acceleration frameworks were used.
  • Workload and baselines: The 1,000-sample benchmark contains shared code contexts followed by four-option questions, mixing cold-start and warm-cache requests across context-length buckets.Contexts range from 163 to 3,400+ tokens; reported buckets include 0–500, 500–1K, and 1K–2K tokens.
  • Accuracy: KVBoost reaches 99.2% exact-match accuracy versus 99.1% for both full recomputation and vLLM prefix caching, with no detectable quality regression.Accuracy remains at or above 98% across the full range of cache reuse ratios.
  • Latency: 4.49× mean TTFT speedup over full recomputation and 14% faster mean TTFT than vLLM prefix caching demonstrate KVBoost’s latency advantage.Mean TTFT is 142.4 ms for KVBoost, compared with 639.1 ms for the baseline and 165.5 ms for vLLM prefix caching.
  • Latency: 4.84× speedup in the 2K+ token bucket shows that KVBoost’s latency benefit grows with context length.Across buckets, speedup over the baseline increases from 3.34× at 0–500 tokens to 4.84× at 2K+ tokens.
  • Cache reuse and memory: KVBoost has a 36.4% mean cache reuse ratio versus 39.5% for vLLM prefix caching, yet it achieves lower overall TTFT through chunk-level reuse and lower overhead.Peak GPU allocation is 6,125.8 MB for KVBoost versus 6,140.6 MB for the baseline, a 14.8 MB reduction.

7 Discussion

KVBoost is most beneficial when shared content occurs at arbitrary positions or cache hits are frequent, while approximate matching requires repair to preserve output quality. Its practical limits include memory planning, RoPE-only compatibility, chunk-size trade-offs, single-GPU scope, probe overhead, and single-task evaluation.

  • When Reuse Helps: Chunk-level reuse benefits arbitrary-position sharing, varying preambles after shared system prompts, and fixed-corpus batch generation with near-100% cache hit ratios.These workloads include bug localization, shared system prompts that are not leading prefixes, and repeated generation over a fixed corpus.
  • Approximate Matching: 99.2% vs. 99.1% accuracy shows approximate-match inference becomes indistinguishable from full recomputation after mandatory CacheBlendRecompute.Approximate matches otherwise risk wrong RoPE rotations and wrong preceding context; setting ρ to 0.25 eliminates observed structured-task differences in practice.
  • Memory Budget: 4 GB KVBoost memory leaves 4 GB for generation KV, supporting approximately 16K-token contexts at float16 on the specified 24 GB GPU.For the 3B-parameter model, 8 GB is occupied by model weights; int8 quantization reduces the KVBoost footprint to 2 GB with negligible quality loss.
  • Limitations: RoPE-only compatibility excludes models using ALiBi or learned absolute position embeddings without a different position-correction mechanism.The limitation is tied to the position-correction method required by the cache reuse design.
  • Limitations: C = 128 balances repair overhead and cache-key hit rates, whereas C = 32 creates many seams and C = 512 produces coarser keys with lower hit rates.Very small chunks increase seam-related repair overhead, while very large chunks reduce cache-hit effectiveness.
  • Limitations: CacheBlend probing can take hundreds of milliseconds beyond 8K cached tokens, while multi-GPU coordination and broader task evaluation remain unimplemented.The current benchmark covers only bug localization with short outputs; proposed mitigations include threshold-based probe activation and future evaluation on longer-form tasks.

8 Conclusion

KVBoost extends KV cache reuse beyond leading-prefix overlap by decoupling content identity from positional identity and repairing boundary artifacts. On Qwen/Qwen2.5-3B bug-localization workloads, it substantially reduces mean TTFT without output-quality regression.

  • Results: 4.49× mean TTFT speedup over full recomputation was achieved on Qwen/Qwen2.5-3B across 1,000 bug-localization samples.The evaluation used realistic workloads where shared content was not confined to a leading prefix.
  • Results: 16% higher mean TTFT performance than vLLM prefix caching was achieved by KVBoost.The comparison was reported on the same Qwen/Qwen2.5-3B evaluation over 1,000 bug-localization samples.
  • Results: 99.2% exact-match accuracy versus 99.1% for both baselines shows no output-quality regression.The accuracy comparison accompanied the reported TTFT results.
  • Core insight: KVBoost decouples content identity from positional identity, enabling KV reuse when shared prompt content appears outside a leading contiguous prefix.This extends KV caching to prompts whose reusable content occurs at arbitrary positions.
  • Core insight: Principled repair for boundary artifacts enables KVBoost to reuse independently positioned content without sacrificing the benefits of KV caching.The repair mechanism addresses artifacts caused by decoupling content identity from positional identity.

A Data Availability

KVBoost is openly available through its MIT-licensed source repository, which also provides benchmark artifacts and scripts. The experiments use publicly available HuggingFace models.

  • A Data Availability: The KVBoost source code is available on GitHub under the MIT License.Repository: https://github.com/pythongiant/kvboost.
  • A Data Availability: Benchmark results, checkpoint files, and figure-generation scripts are included in the repository.These artifacts are located under benchmarks_and_experiments/important/.
  • A Data Availability: All experiments use publicly available models from the HuggingFace Model Hub.

B Author Contributions

S. Unnikrishnan contributed to the paper’s conceptualization, methodology, software, formal analysis, and writing.

  • S. Unnikrishnan handled conceptualization, methodology, software, formal analysis, original-draft writing, and review and editing.

D Ethics Declaration

The research involved no human subjects, personal data, or sensitive data; consequently, no ethics approval was required.

  • The study involved no human subjects, personal data, or sensitive data, so ethics approval was not required.
Loading 2608.21362v1…