PaperlayerView exact revision on arXiv

Source-linked AI summary

Fractional Decay KV-Cache: Ownership-Aware Memory Management for Improved Inference Relevancy in Dialog Systems

Sukanta Ganguly

arXiv:2608.18098v1cs.CLcs.AI

TL;DR

KV caches grow with multi-turn conversations, creating memory pressure and relevance dilution as stale entries accumulate. FD-KVC combines cumulative attention with decaying recency relevance, outperforming H2O by +6.7% on composite alignment across five dialog benchmarks.

  • Problem

    Multi-turn KV caches grow with conversation length, increasing memory pressure and diluting relevance as stale entries accumulate.

  • Method

    FD-KVC combines cumulative attention and temporally decaying recency relevance in a hybrid score governing cache eviction and attention modulation.

  • Results

    Across five multi-turn benchmarks, FD-KVC outperforms H2O on composite alignment, topic adaptation, and topic diversity, including +6.7% composite alignment.

  • Takeaways & Limitations

    FD-KVC provides CPU-only, no-retraining cache management for dialog inference with negligible overhead.

  • Takeaways & Limitations

    Experiments use randomly projected embeddings rather than real transformer representations, so transfer to production settings remains unvalidated.

Abstract

from arXiv · show

Key-value (KV) caching is essential for efficient autoregressive inference in transformer based dialog systems, yet existing strategies treat all cached entries uniformly or apply coarse eviction heuristics that fail to adapt as dialog topics evolve. We propose Fractional Decay KV-Cache (FD-KVC), a novel algorithm that maintains a dual-channel scoring mechanism for each cached KV pair: a cumulative attention channel that tracks aggregate importance (akin to H2O), and a recency-weighted relevance channel governed by temporal decay and reinforcement-inspired updates. The combination enables FD-KVC to both preserve historically important tokens and rapidly adapt when dialog topics shift. An adaptive learning rate driven by an ownership loss function ensures convergence without oscillation. FD-KVC operates entirely on CPU with negligible overhead. Across five diverse multi-turn dialog scenarios with 600 dialogs each, FD-KVC outperforms H2O, the state-of-the-art heavy-hitter baseline, by +6.7% on composite late-turn alignment, with improvements of +127% on topic-shift, +87% on gradual evolution, and +30% on mixed-topic dialogs. FD-KVC adapts to new topics 3.6X faster than H2O and achieves the highest topic diversity (80.6%) across all methods. Ablation studies confirm the contribution of each component.

1 Introduction

FD-KVC addresses KV-cache memory pressure and relevance dilution in multi-turn dialog with a dual-channel scoring framework. It combines cumulative attention for historical importance with temporally decaying, reinforced recency relevance for topic adaptation.

  • Motivation: KV caches grow linearly with conversation length, increasing RAM usage and diluting relevance across cached entries.The cache accumulates representations from all preceding dialog turns, distributing attention over an increasingly large set of entries.
  • Limitations of Existing Methods: Existing sliding-window, heavy-hitter, and learned-compression methods use coarse retention or fail to model temporal dynamics.Cumulative-attention methods such as H2O can retain stale tokens whose early high attention scores persist indefinitely.
  • Proposed Method: FD-KVC assigns each cached KV pair cumulative attention and recency-weighted relevance scores to preserve historically valuable tokens while adapting to current queries.The recency channel decays temporally and is reinforced when entries prove relevant; the combined scores govern eviction decisions.
  • Proposed Method: A reinforcement-inspired update rule uses dynamic reward signals to reinforce the recency channel for contextually relevant cache entries.This mechanism supports continuous relevance tracking rather than binary keep-or-discard decisions.
  • Optimization: An adaptive learning rate driven by a convergence-aware ownership loss function ensures fast convergence.The ownership loss is designed to support convergence without the oscillation mentioned in the paper context.

2 Related Work

Prior KV-cache methods optimize memory, eviction, long-context handling, or learned compression, but they either retain entries, use fixed or attention-based heuristics, or require architectural changes or training. FD-KVC is presented as a drop-in, lightweight, training-free ownership mechanism for cache compression.

  • KV-Cache Optimization: Standard KV-cache optimizations reduce memory or improve allocation and attention computation, but do not address temporal relevance or cache eviction.Multi-query attention and grouped-query attention reduce per-head memory; PagedAttention retains all entries, while FlashAttention does not evict cache entries.
  • Cache Eviction Strategies: Existing cache eviction strategies use sliding windows, attention sinks, cumulative attention, persistence, adaptive attention patterns, or pre-generation position selection.These methods include StreamingLLM, H2O, Scissorhands, FastGen, and SnapKV.
  • Long-Context Methods: Long-context methods modify transformer architecture, whereas FD-KVC operates as a drop-in replacement for the standard cache without retraining.The cited approaches include recurrent memory, retrieval-based unlimited context, and methods targeting long-context factuality.
  • Learned Compression: Learned compression methods such as Gist tokens require specialized compression-module training, while FD-KVC uses a lightweight, training-free ownership mechanism.FD-KVC achieves compression without training specialized compression modules.

3 Fractional Decay KV-Cache

FD-KVC manages cache ownership with complementary long-term and recency-sensitive scores, combining them for adaptive eviction and attention modulation. Its convergence-aware updates preserve stable context while adapting to changing relevance with CPU-efficient per-turn overhead.

  • Dual-channel ownership: FD-KVC assigns each cached key-value pair cumulative attention and recency relevance scores, combining them into a hybrid ownership score for eviction and attention modulation.The cumulative channel preserves historically attended tokens, while the recency channel decays and reinforces relevance as topics change.
  • Dual-channel ownership: The default hybrid weighting is wc = 0.45 and wρ = 0.55, balancing long-term importance against recent relevance.Setting wc = 1 reduces the method to H2O, whereas wρ = 1 uses only decaying relevance.
  • Adaptive convergence: High ownership loss lowers the learning rate to avoid instability, while convergence toward zero restores the initial rate for responsive adaptation.The update converges toward hybrid scores of hi → 0 for eviction and hi → 1 for retention.
  • Eviction and attention: Entries with hi < τ ·maxj hj are evicted, and when the cache exceeds budget B, only the top-B entries by hybrid score remain.Attention modulation uses recency while retaining at least half of each low-recency entry’s unmodulated weight, protecting historically important context.
  • Efficiency: FD-KVC runs once per dialog turn with O(|C|·d) relevance computation and O(|C|) ownership updates and eviction, using CPU-only vectorized operations.The relevance computation is dominated by the O(nt · |C| · d) attention computation, and no GPU or specialized hardware is required.

4 Experimental Setup

The experiments use a controlled synthetic multi-turn dialog benchmark with five 600-dialog scenarios, shared cache pressure, and metrics targeting topic tracking, retention, diversity, adaptation, and latency. Comparisons use a 56-token cache budget across standard and attention-based baselines.

  • Benchmark scenarios: Five benchmark scenarios contain 600 dialogs each, including topic shift, topic return, 5-topic complex, and gradual evolution settings.These scenarios test cache adaptation, long-term retention, multi-topic coverage, and smooth topic changes.
  • Baselines and cache budget: 56 tokens (≈1.75 turns) is the shared cache budget for all baselines, which use the same projection matrices.The baselines are Standard (FIFO), Sink+Window, and H2O.
  • Evaluation metrics: Late-Turn Alignment (LateAl) is the primary metric, measuring cosine similarity between attention output and the projected ground-truth topic vector over the last 3 turns.It evaluates how well the cache tracks the current topic under sustained pressure.
  • Evaluation metrics: Late Topic Retention (LateRet), Late Topic Diversity (LatDiv), and Adaptation Speed quantify current-topic coverage, multi-topic representation, and turns needed to reach 80% retention after a shift.Higher LatDiv is better, while lower Adaptation Speed is better.
  • Evaluation metrics: Latency is measured as wall-clock time per turn in milliseconds.This captures the runtime cost of each cache strategy.

5 Results

FD-KVC improves late-turn alignment over H2O, especially when dialog topics change, while adapting faster and preserving greater topic diversity. These gains involve tradeoffs in Topic Return performance, latency, and retention.

  • FD-KVC achieves +6.7% composite improvement over H2O, with +127% on Topic Shift, +87% on Gradual Evolution, and +30% on the Mixed benchmark.The improvements are strongest in scenarios involving topic change.
  • H2O achieves +64% over FD-KVC on Topic Return, as non-decaying cumulative scores preserve tokens through off-topic gaps.This reflects a tradeoff associated with FD-KVC’s temporal decay.
  • 3.6× faster adaptation lets FD-KVC reach 80% new-topic retention in 4.5 turns after a shift, versus H2O’s 16.0 turns.H2O’s stale cache retains old-topic tokens with high accumulated scores, whereas FD-KVC uses recency decay to transition.
  • FD-KVC reaches 80.6% composite topic diversity, surpassing FIFO (47.6%), Sink+Window (73.3%), and H2O (70.0%).Its multi-topic retention supports dialogs where users reference earlier topics.
  • FD-KVC latency is 0.105 ms/turn versus FIFO’s 0.027 ms, while retention reaches 70.1%, above H2O’s 63.3% but below FIFO’s 91.4%.The 0.078 ms increase is described as negligible relative to typical LLM inference times of 50–500 ms per token.
  • Paired tests show consistent gains but limited individual-benchmark significance: Topic Shift p=0.19, Mixed p=0.57, and Gradual p=0.39.The composite improvement and adaptation-speed analysis provide complementary evidence.

6 Ablation Study

The ablation study identifies balanced cumulative weighting and recency decay as important for trading off long-term memory, topic-shift adaptability, and retention. It selects γ = 0.80 for best alignment and γ = 0.88 for production use.

  • Cumulative weighting: The cumulative-weight sweep peaks around wc = 0.3, while extreme values reduce performance by sacrificing either long-term memory or adaptability.Too little cumulative weight loses long-term memory; too much resembles H2O and loses adaptability.
  • Decay rate: The best alignment occurs at γ = 0.80, whereas higher decay rates degrade retention as old recency scores compete with new ones.Production FD-KVC uses γ = 0.88 to balance shift adaptation with return retention.
  • Channel ablation: Removing either scoring channel through extreme wc values degrades performance, confirming that both channels contribute to the alignment–retention tradeoff.The two-channel ablation is summarized as an alignment–retention tradeoff across the sweeps.

7 Discussion

FD-KVC separates long-term importance from recent relevance, allowing it to adapt faster than H2O while retaining historically important tokens. Its design is supported by reinforcement-learning-inspired updates and practical CPU-only deployment without retraining.

  • Why Dual-Channel Scoring Works: 3.6× faster adaptation follows from FD-KVC’s decaying recency channel, while its cumulative channel provides a stable floor against premature eviction.H2O’s cumulative attention score only increases, creating stale-cache problems; FD-KVC instead separates long-term importance from recent relevance.
  • Why Dual-Channel Scoring Works: 80.6% vs H2O’s 70.0% diversity reflects FD-KVC’s ability to preserve historical importance while adapting to recent relevance.The cumulative channel prevents premature eviction, whereas the recency channel decays old relevance.
  • Reinforcement Learning Connection: The recency update can be interpreted as a policy gradient using relevance r_i as reward, with the adaptive learning rate providing variance reduction.This reinforcement-learning connection distinguishes FD-KVC’s principled updates from ad-hoc scoring heuristics.
  • Practical Deployment: 0.105 ms/turn is FD-KVC’s CPU-only deployment cost as a drop-in cache manager requiring no model retraining.Ablations report peak alignment at w_c = 0.3 and best alignment for faster decay, γ ≈0.80, on Mixed, 200 dialogs.

8 Conclusion

FD-KVC combines cumulative attention tracking with temporally decaying recency relevance to address stale caches in dialog-system inference. Across five multi-turn benchmarks, it outperforms H2O, adapts faster to topic shifts, and achieves the highest topic diversity with negligible CPU overhead.

  • Conclusion: +6.7% composite alignment: FD-KVC outperforms H2O across five multi-turn benchmarks.FD-KVC also adapts 3.6× faster to topic shifts and achieves 80.6% topic diversity.
  • Conclusion: FD-KVC uses dual-channel scoring that combines cumulative attention tracking with temporally decaying recency relevance.This design addresses the stale cache problem inherent in existing heavy-hitter methods.
  • Conclusion: FD-KVC runs on CPU with negligible overhead and requires no additional infrastructure.
  • Future work: Future work will extend FD-KVC to multi-head attention and evaluate it on real-world dialog tasks including MultiWOZ and SGD.
  • Future work: Future work will explore retrieval-augmented generation pipelines where ownership scores inform cache priority for retrieved passages.

9 Limitations · A Hyperparameter Sensitivity

The evaluation is limited by synthetic embeddings, proxy metrics, single-head attention, computational overhead, and unresolved adaptation–persistence and channel-weight tradeoffs. FD-KVC also uses seven task-sensitive hyperparameters, with Table 7 documenting default values and explored ranges.

  • 9 Limitations: Synthetic 64-dimensional randomly projected embeddings limit individual-benchmark statistical significance and require validation with real LLaMA- or GPT-family representations.The controlled setup may not establish transfer to production settings.
  • 9 Limitations: 39%: FD-KVC underperforms H2O on Topic Return because temporal decay prevents recovery of tokens when topics recur.Higher wc or auxiliary retrieval may be needed when distant-context recall matters.
  • 9 Limitations: Single-head evaluation leaves the optimal multi-head scoring design unresolved, requiring per-head scoring or a shared strategy.Different attention heads may focus on different aspects of context.
  • A Hyperparameter Sensitivity: Seven hyperparameters make the optimal FD-KVC configuration potentially task-dependent despite stable ablation performance; automated tuning could support deployment.Table 7 provides the default hyperparameter values and explored development ranges.
  • 9 Limitations: 0.105 ms/turn: FD-KVC latency is 3.9× FIFO because of dual-channel scoring and hybrid eviction.The overhead is negligible relative to LLM decoding but may matter when applied simultaneously at every transformer layer.
  • 9 Limitations: Alignment, retention, and diversity are proxy metrics, so end-to-end dialog evaluation of response quality, slot filling, and user satisfaction remains needed.Such tasks would strengthen the conclusions.
  • 9 Limitations: Fixed wc and wρ weights cannot shift between recency during rapid topic changes and cumulative importance during stable phases.An adaptive weighting mechanism could yield further improvements.

B Implementation Details

FD-KVC was implemented as a CPU-efficient NumPy system with lightweight dual-score storage and vectorized relevance and eviction operations. Experiments ran on a single Apple M-series CPU core, with each benchmark completing in under 60 seconds.

  • Execution environment: Under 60 seconds: each benchmark of 600 dialogs × 4 methods completed on a single Apple M-series CPU core.The implementation uses NumPy 1.24+ for vectorized operations.
  • Data representation: 8|C| bytes: cumulative and recency scores were stored as two 1D float32 arrays.Original embeddings were retained alongside projected keys and values for embedding-space relevance computation.
  • Vectorized computation: Vectorized cosine similarity: relevance computation used batch matrix operations.This implementation choice supports efficient embedding-space relevance calculation.
  • Eviction: Efficient top-B selection: eviction applied np.argsort to the hybrid score.The hybrid score combines the cumulative and recency channels described in the implementation.
Loading 2608.18098v1…