PaperlayerView exact revision on arXiv

Source-linked AI summary

EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments

Minsoo Kim, Arnav Kundu, Han-Byul Kim, Richa Dixit, Minsik Cho

arXiv:2509.17396v4cs.CL

TL;DR

LongConvQA over extended histories is challenging under resource constraints, while common memory approaches do not bound inference memory and many KV methods evict only after full-context prefill. EPICACHE uses block-wise prefill and episode-specific KV compression under a fixed budget. Across LongConvQA benchmarks, it improves accuracy while reducing cache-related resource costs.

  • Problem

    LongConvQA needs memory-bounded inference over multi-session histories, but retrieval banks leave memory unbounded and most KV compression waits until full-context prefill.

    Source: §1 · p. 1

  • Method

    EPICACHE uses block-wise prefill and episode-specific KV compression, with sensitivity-aware layer-wise budgets to preserve topic-relevant history under a fixed memory budget.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    EPICACHE achieves up to 30% higher LongConvQA accuracy than recent baselines, comparable-to-full-KV accuracy under 4–6× compression, peak-memory reduction up to 3.5×, and decoding-latency reduction up to 2.4×.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across multiple LongConvQA benchmarks, EPICACHE demonstrates long-term interaction under strict resource constraints as a practical step toward on-device conversational AI.

    Source: §6 · p. 9

    Evaluation on more challenging benchmarks tracking implicit preferences and information updates remains future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Modern large language models (LLMs) extend context lengths to millions of tokens, enabling coherent, personalized responses grounded in long conversational history. However, the Key-Value (KV) cache grows linearly with the extended dialogue history, causing the model's memory footprint to quickly exceed device limits. While recent KV cache compression methods attempt to reduce memory usage, most apply cache eviction after processing the entire context, incurring unbounded peak memory usage. Additionally, query-dependent eviction narrows the cache semantics to a single query, leading to failure cases in multi-turn conversations. In this paper, we introduce EpiCache, a training-free KV cache management framework for long conversational question answering (LongConvQA) under fixed memory budgets. EpiCache bounds cache growth through block-wise prefill and preserves topic-relevant context via episodic KV compression, which clusters conversation history into coherent episodes and performs episode-specific KV cache eviction. Across three LongConvQA benchmarks (LongMemEval, Realtalk, and LoCoMo), EpiCache improves accuracy by up to 30%, achieves near full-cache accuracy under 4-6x compression, and reduces latency and peak memory by up to 2.4x and 3.7x, respectively.

1. Introduction

LongConvQA calls for answers grounded in extended conversational histories, but existing approaches do not provide bounded inference memory while preserving information across turns. EPICACHE combines block-wise prefill and episodic cache compression, with reported accuracy and efficiency gains under fixed budgets.

Motivation

  • LongConvQA requires answering sequences of questions from multi-session dialogue, while common retrieval memory banks use repeated API-based LLM calls and do not bound inference memory.These constraints limit their applicability in resource-constrained environments.Source: §1 · p. 1
  • After only 30 sessions, LLaMA3.2-3B's KV cache exceeds 7GB, illustrating the memory pressure of long dialogues on constrained devices.The paper notes that this exceeds the model-parameter size.Source: §1 · p. 1

Limitations of prior approaches

  • Most compression evicts after full-context prefill, so peak memory scales linearly with input; query-dependent eviction may also discard evidence needed in later turns.Source: §1 · pp. 1–2

Approach

  • EPICACHE combines block-wise prefill, evicting after each block, with clustering into coherent episodes and topic-specific cache compression under constrained memory.The framework is training-free.Source: §1 · p. 2

Results

  • Up to 30% higher LongConvQA accuracy than recent baselines; accuracy comparable to full KV under 4–6× cache compression; peak memory reduced by up to 3.5× and decoding latency by up to 2.4×.Source: §1 · p. 2

2. Background

The background formulates LongConvQA as multi-turn answering over shared dialogue and reviews why cache management must balance memory bounds with answer quality. It contrasts post- and block-prefill eviction, then describes attention-guided scoring and patched prompts as a way to guide eviction.

2.1. Long Conversational QA Formulation

  • LongConvQA asks an assistant to answer a sequence of queries grounded in shared dialogue history; the objective is all-query answers from budgeted compressed KV comparable to full KV.Source: §2.1 · p. 2

2.2. KV Cache Management: Post vs. Block Prefill

  • Post-prefill eviction leaves peak memory linear in input; block prefill keeps entries within M + Mblock, but KVzip's LongConvQA accuracy falls sharply across budgets in block mode.Figure 1 contrasts peak GPU memory with input length and LongConvQA accuracy under post- versus block-prefill eviction.Source: §2.2 · p. 2, Figure 1 · p. 3

2.3. Attention-guided KV Cache Compression

  • Attention-guided compression scores tokens by their attention from query tokens, then evicts lower-scoring entries to meet the cache budget.A patched prompt supplies the scoring queries and is not retained in the cache.Source: §2.3 · p. 3
  • Patched-prompt content affects accuracy: semantically aligned utterances in the top 10% nearly match Exact-Question performance, while weaker alignment degrades accuracy.Exact-Question uses the future query as the prompt, but is infeasible because future queries are unknown at compression time and query-dependent compression requires re-prefilling the dialogue for each question.Source: §2.3 · p. 3, Figure 2 · p. 3

3. Method

EpiCache organizes conversation history into topic-specific episodic caches and matches each query to a relevant cache, while block-wise prefill bounds cache growth. It also allocates cache budget by layer sensitivity to reduce the divergence between block-prefill and full-cache answer predictions.

3.1. Episodic KV Cache Management

  • Conversation segments are clustered into coherent episodes, whose centroid-closest segments serve as patched prompts for constructing episode-specific KV caches under budget M.At decoding, each query is matched to the closest episode centroid and generation uses that episode’s cache.Source: §3.1 · pp. 4–5
  • Retrieved caches are extended with current-turn KV pairs, new episodes are created for queries dissimilar to existing centroids, and caches are rebuilt when they exceed M + Mblock.Rebuilding reclusters the accumulated episode history and uses an updated representative segment as the patched prompt.Source: §3.1 · p. 5

3.2. Sensitivity-aware Layer-wise KV Budget Allocation

  • Layer sensitivity is measured from the deviation between full-mask and block-prefill Key states, and more sensitive layers receive larger cache budgets.The method simulates block prefill with a custom mask, defines sensitivity as 1 − σℓ, and redistributes the global budget according to layer sensitivity scores.Source: §3.2 · p. 5
  • –0.80 KL-divergence shift relative to uniform allocation was reported for sensitivity-aware budgeting, with negative values indicating closer alignment to full-KV answer predictions.The paper reports that this reduction is reflected in improved LongConvQA accuracy.Source: Figure 4 · p. 5, §3.2 · p. 6

4. Experiments

The experiments compare EPICACHE with retrieval and KV-compression baselines across LongConvQA benchmarks, then examine multi-episode reasoning, design choices, robustness, and efficiency. EPICACHE performs strongly under tight cache budgets and reduces memory and latency in long-term conversational inference.

4. Experiments

  • Evaluations use Realtalk, LoCoMo, and LongMemEval with Qwen3, LLaMA3.1/3.2, and Qwen2.5, comparing retrieval-based and KV-compression baselines.The retrieval baselines are RAG-2K and RAG-Episodic; compression baselines include KVzip, OracleKV, StreamingLLM, SnapKV, KeyDiff, and InfiniPot.Source: §4 · p. 6

4.1. Main Results

  • Up to 30 absolute score improvement over compression baselines on Realtalk; across three LongConvQA benchmarks and tested budgets, EPICACHE performs best.The Realtalk comparison covers four LLMs and reports convergence to Full KV performance as the budget increases.Source: §4.1 · pp. 6–7, Figure 6 · p. 7
  • Across NarrativeQA, HotpotQA, MuSiQue, and QMSum, EPICACHE beats all baselines on Qwen3-4B and 8B, with largest gains at 2–4K budgets.The evaluation averages results over four long-form document QA tasks and fixes the number of episodes at E=4.Source: §4.1 · p. 7, Figure 7 · p. 7

4.2. EPICACHE Analysis

  • EPICACHE stays closer to Full KV across Src 1/2/3+ groups, while Realtalk Cross and Temporal scores reach 50.8 / 55.7 versus RAG’s 31.9 / 22.4.The analyses group queries by source segments and by query type; EPICACHE’s caches are built through block-wise prefill over the full conversation before compression.Source: §4.2 · pp. 7–8, Figure 8 · p. 7
  • On Realtalk, utterance-level segmentation beats word- and token-level alternatives; Qwen3-Embedding performs best, and E=4 scores 49.8 versus KVzip’s 36.0.A 0.6B encoder is sufficient with negligible overhead relative to the base LLM; moderate allocation sharpness of 2–4 performs best, while 8 hurts accuracy.Source: §4.2 · p. 8, Table 2 · p. 8
  • Across LoCoMo and Realtalk, stochastic and calibration conditions yield standard deviations below 0.5; one calibration sample matches 100 across tested domains.The calibration domains are dialogue history, code repository, and long-form narrative; the results indicate layer sensitivity is largely model-dependent rather than data-dependent.Source: §4.2 · p. 8, Figure 9 · p. 8

4.3. Efficiency Analysis

  • For a 90K-token history and 300 subsequent interactions, EPICACHE uses 9.6 GB peak GPU memory versus 36.3 GB for Full KV, and takes 545.4 s total.That total is nearly 18× faster than Full KV and approximately 2× faster than prefix caching, with comparable QA accuracy of 46.2 vs. 45.6.Source: §4.3 · pp. 8–9, Table 3 · p. 9

5. Related Work

Related work addresses KV eviction and selective retrieval to reduce cache or decoding costs. The cited approaches differ in their eviction signals, training requirements, and whether they assume unconstrained prefill memory.

5. Related Work

6. Conclusion

EPICACHE combines block-wise prefill, episodic clustering, and sensitivity-aware budget allocation under a fixed memory budget. The paper identifies harder preference-tracking and information-update benchmarks as future evaluation targets.

6. Conclusion

  • EPICACHE combines block-wise prefill, episodic clustering, and sensitivity-aware budget allocation to preserve topic-relevant context under a fixed memory budget.Across multiple LongConvQA benchmarks, it substantially outperforms existing compression methods.Source: §6 · p. 9
  • Future evaluation targets harder benchmarks requiring implicit user-preference tracking and handling information updates over time.The authors also identify improved conversational embeddings and adaptive episode-count selection as future work.Source: §6 · p. 9

A.1. Dataset

The appendix describes the three LongConvQA datasets and evaluation settings, then details compression baselines and EPICACHE’s offline preparation and online routing.

A.1. Dataset

  • Realtalk contains 10 real-world conversations spanning 16–21 days, while LoCoMo contains 10 human-machine conversations of up to 35 sessions and around 300 turns.Realtalk evaluation covers multi-hop, temporal reasoning, and commonsense; LoCoMo has five subtasks and uses F1 scoring.Source: §A.1 · p. 14
  • Adversarial LoCoMo results are excluded because compression can inflate scores through over-prediction of unanswerability rather than improved answer quality.The paper defers a more thorough study of unanswerability detection under compression to future work.Source: §A.1 · p. 14
  • LongMemEval histories are session-stacked into chronologically consistent contexts from 20K to 100K tokens while preserving the benchmark’s question-type distribution.QA pairs are sampled by task-type proportions, with their corresponding evidence sessions assembled into histories.Source: §A.1 · p. 14

A.2. KV Cache Compression Baseline Setup

  • Compression baselines span static retention, attention-based eviction, and similarity-based selection, adapted to block prefill for comparison.Attention-based methods share a scoring formulation, and eviction methods use head-wise non-uniform token selection.Source: §A.2 · pp. 14–16

A.3. EPICACHE Setup

  • EPICACHE segments and embeds conversation history, clusters segments into episodes, and uses each episode’s centroid and medoid segment as its representation and patched prompt.The offline segmentation and clustering stage completes within one minute.Source: §A.3 · p. 16
  • EPICACHE measures per-layer sensitivity, allocates cache budgets accordingly, and builds a compressed cache for each episode using block-wise prefill.At query time, it routes each query to the most similar episode centroid and loads that episodic cache for decoding.Source: §A.3 · p. 16
  • The block size trades temporary peak memory against coverage speed: larger blocks cover the conversation faster but raise peak memory.The evaluated Mblock settings are 128, 512, 1024, and 2048.Source: §A.3 · p. 16

B.1. Conversation Clustering Analysis

The clustering analysis illustrates that conversation segments form interpretable topical episodes, with medoid segments representing each cluster.

B.1. Conversation Clustering Analysis

  • K-means clusters of four-utterance segments group into coherent topics such as games, movies, literature, and weather.Representative medoid segments make each episode interpretable and illustrate how episode partitioning can support episodic KV cache compression.Source: §B.1 · p. 16

B.2. block-wise prefill Sensitivity Analysis

The sensitivity analysis compares internal states under full and block-wise prefill masks. Key-state deviations distinguish layers most clearly and motivate their use for budget allocation.

B.2. block-wise prefill Sensitivity Analysis

  • The analysis compares Key and Value cosine similarities and layer-output L2 distances across layers under full causal and block-wise masks.It uses LoCoMo conversation histories; the figure’s shaded regions indicate variance across input samples.Source: §B.2 · pp. 16–17, Figure A2 · p. 17
  • Key-state cosine similarities differentiate layers more clearly than Value-state similarities or layer-output L2 distances, motivating Key deviation for budget allocation.Value similarities remain consistently low, while output distances show monotonic error accumulation rather than meaningful layer variation.Source: §B.2 · p. 17
  • The authors leave further analysis of why Key, Value, and layer-output trends differ for future work.Source: §B.2 · p. 17

C.1. Comprehensive LongConvQA Results

The evaluation covers multiple models, benchmarks, and cache budgets, with a separate study scaling conversation histories to 100K tokens. EPICACHE outperforms baselines at matched budgets in that scalability evaluation.

C.1. Comprehensive LongConvQA Results

  • The comprehensive evaluation compares EPICACHE with baselines across cache budgets and four model variants, with numerical results reported in Tables A4–A6.The evaluations cover Realtalk, LoCoMo, and LongMemEval.Source: §C.1 · p. 17

C.2. Memory Scalability Evaluation.

  • Across context lengths up to 100K tokens, EPICACHE achieves higher accuracy than baselines at the same memory budget, and its accuracy approaches full KV as budget increases.The comparison uses LongMemEval and evaluates InfiniPot and KVzip (M=6K) against EPICACHE with four episodes and M=6K–24K.Source: §C.2 · p. 18, Figure A3 · p. 18

D. Future Work

Future work targets more coherent episode construction and broader memory-management capabilities for long-term conversational QA.

D. Future Work

  • Better conversational embeddings that capture turn-level context and inter-utterance dependencies could yield more coherent episode boundaries.Episode-boundary quality depends heavily on the underlying segment representations.Source: D. Future Work · p. 18
  • Adaptively determining the number of episodes could improve scalability across dialogue lengths and domains.Source: D. Future Work · p. 18
  • Future memory management should address implicit preferences, information updates and deletions, persona consistency, and combining text-level organization with KV cache compression.The paper identifies these as richer challenges and a promising direction for efficient conversational agents on resource-constrained devices.Source: D. Future Work · p. 18
Loading 2509.17396v4…