Source-linked AI summary
KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
Chuangtao Chen, Grace Li Zhang, Xunzhao Yin, Cheng Zhuo, Bing Li, Ulf Schlichtmann
TL;DR
Context-dependent KV caches become stale when documents are reused with different prefixes and positions, forcing recomputation that raises serving cost. KV Packet wraps frozen document caches with trainable boundary adapters learned by self-supervised distillation. The method reduces inference-time FLOPs by 5–6 orders of magnitude while matching state-of-the-art accuracy on retrieval and reasoning benchmarks, but its generalization remains bounded by the evaluated domains and document settings.
Problem
Context-dependent KV caches cannot be directly reused across changing prefixes and positions, limiting modular document caching for multi-document retrieval.
Method
KV Packet wraps frozen document caches with lightweight trainable Headers and Trailers, optimized by self-supervised distillation while base-model weights remain frozen.
Results
KV Packet reduces inference-time FLOPs by 5–6 orders of magnitude while matching state-of-the-art accuracy on retrieval and reasoning benchmarks.
Takeaways & Limitations
The recomputation-free design integrates with existing KV compression techniques and enables direct cache composition at serving time.
Takeaways & Limitations
Generalization remains open for highly out-of-distribution domains, broader model families, and dependent document chains such as multi-step reasoning traces.
Abstract
from arXiv · showhide
Large Language Models (LLMs) rely heavily on Key-Value (KV) caching to minimize inference latency. However, standard KV caches are context-dependent: reusing a cached document in a new context requires recomputing KV states to account for shifts in attention distribution. Existing solutions such as CacheBlend, EPIC, and SAM-KV mitigate this issue by selectively recomputing a subset of tokens; however, they still incur non-negligible computational overhead (FLOPs) and increased Time-to-First-Token (TTFT) latency. In this paper, we propose KV Packet, a recomputation-free cache reuse framework that treats cached documents as immutable ``packets'' wrapped in light-weight trainable soft-token adapters, which are trained via self-supervised distillation to bridge context discontinuities. Experiments on Llama-3.1 and Qwen2.5 demonstrate that the proposed KV Packet method achieves near-zero FLOPs and lower TTFT than recomputation-based baselines, while retaining F1 scores comparable to those of the full recomputation baseline.
1 Introduction
KV caching is difficult to reuse across changing retrieval contexts because independently cached documents encode incomplete attention context. KV Packet addresses this with immutable document caches wrapped in trainable adapters, avoiding inference-time recomputation while preserving performance and compatibility with compression.
- Motivation: Context-dependent KV states make independently cached documents invalid when concatenated with other retrieved chunks.Each cache reflects attention over an incomplete prefix and misses cross-document dependencies.
- KV Packet: KV Packet wraps frozen document caches with trainable soft-token adapters, eliminating model parameter modification and inference-time cache recomputation.The adapters are trained through self-distillation without human-labeled data.
- Results: KV Packet reduces computational overhead by approximately 4 orders of magnitude versus CacheBlend and EPIC while maintaining competitive performance across tasks.The paper presents this as its efficiency and performance result.
- Compatibility: KV Packet natively supports existing KV compression techniques, whereas recomputation-based methods struggle with unstructured token preservation across layers.This compatibility is presented as a consequence of the recomputation-free design.
2 Background and Related Work
KV caches depend on both preceding context and absolute positions, limiting modular reuse in multi-document retrieval. Prior approaches address this through positional realignment, model modification, or selective recomputation, but incur practical costs in deployment, latency, computation, or generality.
- KV-cache dependencies: A precomputed KV cache is valid only with the same prefix and position, so shifting a document by one token can invalidate the entire cache.This prefix and positional dependency prevents modular reuse in dynamic multi-document RAG.
- Positional alignment: RoPE enables each cached key to be realigned to a shifted position through a single rotation at near-zero computational cost.Positional alignment alone does not resolve contextual staleness.
- Contextual dependency: Naive concatenation creates isolated attention blocks because independently computed document representations lack semantic influence from the global prefix.The resulting attention map differs from continuous full-context attention.
- Related work: Model-modification methods can require substantial resources, multiple served models, or risk catastrophic forgetting from changing base-model weights.These drawbacks distinguish them from KV Packet’s frozen-base approach.
- Related work: Selective recomputation repairs cache staleness but adds forward passes, token-selection computation, substantial cumulative FLOPs, and architecture-specific engineering.These costs complicate adaptation across model architectures.
3 Methods
KV Packet treats each document as a frozen cache enclosed by learned boundary adapters that absorb concatenation artifacts. The adapters are trained by self-supervised distillation so packets can be concatenated directly at serving time without recomputation.
- Design motivation: KV Packet addresses naive-concatenation degradation by targeting boundary artifacts, including disrupted attention sinks and abrupt token-distribution shifts.The method hypothesizes that these artifacts contribute substantially to performance loss.
- KV Packet formulation: A KV Packet consists of a frozen document cache wrapped by learnable Header and Trailer adapter vectors.The packet representation places the Header before document embeddings and the Trailer after them.
- Inference: Packets are precomputed offline and directly concatenated at serving time, with adapters trained as universal delimiters that avoid cross-document recomputation.The adapters are global and document-independent, and their storage cost is O(1) with respect to corpus size.
- Training: Self-supervised distillation trains the adapters by matching packet-based student behavior to full-context teacher behavior.The model generates a reference continuation with standard causal attention and compares student and teacher token distributions.
- Training: Only the small adapter tensors receive gradients while the base model and document embeddings remain frozen, requiring no human-annotated data.The design keeps memory and FLOPs below fine-tuning methods and avoids catastrophic forgetting by construction.
4 Experiments
KV Packet is evaluated across retrieval and reasoning datasets, language models, efficiency metrics, compression settings, cross-domain transfer, and attention behavior. It generally preserves generation quality while sharply reducing computational and latency costs relative to recomputation-based cache reuse.
- Experimental Setup: Experiments use Llama-3.1-8B-Instruct and Qwen-3-4B-Instruct across Needle-in-a-Haystack, Biography, HotpotQA, and MusiQue, spanning retrieval and multi-step reasoning.All experiments run on one NVIDIA A100 GPU with eight adapter tokens and the stated training configuration.
- Generation Quality: KV Packet achieves comparable F1 on most tasks and substantially outperforms No Recompute, while Qwen on MusiQue shows a gap versus Full Recompute.Despite that gap, the paper reports a favorable FLOPs–TTFT trade-off for this setting.
- Computational Efficiency: 6.50 × 10^-6 to 1.04 × 10^-5 FLOPs are reported for KV Packet, 5–6 orders of magnitude below Full Recompute and matching No Recompute.The FLOPs metric includes cache preparation before generation; KV Packet requires only RoPE position realignment.
- Latency: 1.36× and 3.3× speedups are reported on Llama Biography and HotpotQA, while TTFT reductions reach 19.45× on Needle-in-a-Haystack and 5.81× on MusiQue.KV Packet is described as having TTFT similar to No Recompute and slightly above No Cache.
- Compatibility with KV Compression: KV Packet is more robust than the baseline under random pruning across compression rates of 10%–50% and five compression methods.The evaluated configurations are KVPacket Normal, KVPacket Keep Filler, and Single Cache.
- Cross-Domain Generalization: The Universal wrapper is the most stable cross-domain configuration, matching 0.95 peak Biography performance and reaching 0.43 on MusiQue.Specialized adapters perform well in-domain but transfer less reliably to tasks with different linguistic or reasoning demands.
- Attention Analysis: KV Packet’s adapters receive higher attention than document tokens, absorbing sink mass and restoring a smoother distribution over document content.The attention analysis averages across layers, query tokens, and 50 randomly selected samples per dataset.
5 Conclusion
KV Packet provides recomputation-free, context-independent KV caching by wrapping immutable document caches with lightweight trainable adapters and self-supervised distillation. It integrates with KV compression, reduces inference-time FLOPs by 5–6 orders of magnitude, and matches state-of-the-art accuracy, while leaving several generalization and evaluation boundaries open.
- Contributions: KV Packet wraps immutable document caches with lightweight Headers and Trailers to address boundary artifacts while keeping base model weights frozen.The adapters are trained with a self-supervised distillation objective without human-labeled data.
- Efficiency and compatibility: KV Packet integrates with existing KV compression techniques and reduces inference-time FLOPs by 5–6 orders of magnitude while matching state-of-the-art accuracy.The paper contrasts this compatibility with recomputation-based methods, which cannot fundamentally match it under the stated setting.
- Limitations and future work: Generalization to highly out-of-distribution domains, additional model families, and dependent document chains remains an open question.The current evaluations cover a limited set of model families, and the method targets largely independent retrieved documents.