PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Should the KV Cache Live? Placement Policies Across GPU, CPU, and SSD for Long-Lived Sessions

Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly

arXiv:2609.16215v1cs.AI

TL;DR

Long-lived sessions can accumulate KV state beyond scarce GPU HBM, making placement across GPU, CPU, and SSD an important policy question. The paper evaluates tier placement and prefetch choices across workloads with different reuse patterns. Tiering’s capacity gains are policy-independent, while placement mainly affects data movement and latency-related outcomes.

  • Problem

    The open question is which KV blocks should occupy each tier and when to move or prefetch them, given workload-specific reuse and large tier bandwidth gaps.

    Source: §I · p. 1

  • Method

    The paper builds a configurable tiered placement layer and compares placement, promotion, eviction, and prefetch policies across workloads.

    Source: §I · p. 1

  • Results

    Tiering’s capacity gains are policy-independent (73.02× more sessions per GPU; 62.04× lower cost per session), while placement affects workload-specific data movement.

    Source: §VI · p. 8, §VII · p. 8

  • Takeaways & Limitations

    Recency is recommended for concurrent chat and reuse-frequency for repeated document QA; for one isolated long chat, eviction order is identical across policies.

    Source: §VII · p. 8

    Synthetic reuse patterns make policy separations cleaner than real conversation traces, where reuse is lower and margins compress.

    Source: §VI · p. 8

Abstract

from arXiv · show

GPU high bandwidth memory is scarce and expensive, and KV caches consume much of it as chats, agent loops, and document question answering accumulate state. Systems such as Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore extend GPU memory with CPU DRAM and SSD. The harder question is which blocks belong in each tier, when to move or evict them, and whether prefetching helps. We study these choices in a discrete event simulator spanning GPU HBM, CPU DRAM, and SSD, calibrated against a random forest execution time predictor. We compare recency, reuse frequency, predicted reuse, and an EWMA predictor with prefetch lookahead across chat, agent, and document question answering workloads. Tiering supports 73.02 times more concurrent sessions per GPU and lowers cost per session by 62.04 times. These gains come from tier capacities of 1 plus 8 plus 64, not placement policy. Decode is compute bound at batch size one in our setup, so placement barely affects throughput. It mainly changes PCIe migration traffic and time to first token. Recency produces 2.30 times less migration traffic than reuse frequency for chat. Reuse frequency performs best for agents and document question answering. The existing predicted reuse policy is byte identical to recency, making its agent recommendation effectively recency. A genuine EWMA predictor changes behavior but still ranks behind reuse frequency on the workloads prediction was expected to help. Prefetching does not justify its bandwidth cost. Across the policy and cache size grid, even an oracle with knowledge of future requests never beats no prefetch on migration traffic. Workload specific placement can reduce data movement, but the predicted reuse and prefetch recommendations are not supported as implemented.

I. INTRODUCTION

Long-lived sessions accumulate KV state beyond scarce, expensive GPU HBM, while existing systems can extend storage across CPU DRAM and SSD. The unresolved question is how to place, move, and prefetch blocks across workloads with different reuse patterns.

I. INTRODUCTION

  • Long-lived concurrent sessions can exceed HBM capacity, leading systems to recompute prefill on resume or cap concurrency.Both alternatives waste compute and money.Source: §I · p. 1
  • The policy problem is deciding which blocks belong in each tier and when to promote, demote, evict, or prefetch them.Tier bandwidth and latency gaps mean poor placement can waste HBM or stall decoding.Source: §I · p. 1
  • The best placement depends on workload reuse: document QA reuses prefixes, agents revisit recent tool outputs, and chats grow over time.The paper treats policy choice as an open question distinct from the tiering mechanism.Source: §I · p. 1
  • The study builds a configurable three-tier placement layer and compares policies on chat, agent, and document-QA workloads.Its contributions include measuring concurrent sessions per GPU and cost per session.Source: §I · pp. 1–2

II. BACKGROUND AND RELATED WORK

The KV cache grows with context length and is managed in fixed blocks, making physical placement across memory tiers distinct from techniques that shrink the cache. Prior systems establish tiered KV storage, while general placement policy remains an open question.

A. The KV cache and its cost

  • A context of length T caches 2LHdT scalars, and fixed-size PagedAttention blocks provide the granularity for tier placement.Quantization and eviction shrink the cache; placement decides where blocks physically live.Source: A. The KV cache and its cost · p. 2

B. Tiered KV storage systems

  • Existing systems offload or reuse KV across CPU, DRAM, and SSD, but leave general placement policy and its workload dependence open.Examples include Mooncake [1], LMCache [2], FlexGen [3], InfiniGen [4], AttentionStore [5], and CacheBlend [7].Source: B. Tiered KV storage systems · p. 2

C. Prefix and cross-request reuse

Reuse makes tiering useful, but workloads differ in reuse structure and memory tiers differ sharply in bandwidth. The paper studies workload-specific placement across three tiers, where unfetched blocks can stall decode.

C. Prefix and cross-request reuse

D. The bandwidth wall

  • HBM, DRAM, and SSD differ by orders of magnitude in bandwidth, so fetching a hot SSD block mid-decode can cost more than recomputing it.A cold fetch stalls decode unless prefetch hides it.Source: D. The bandwidth wall · p. 3

E. Our position

  • The study takes offload as given and compares promotion, eviction, and prefetch policies across workloads with different reuse structures.It reports a policy guide and the gain in sessions per GPU.Source: E. Our position · p. 3

A. The tiered layer

  • Each fixed-size KV block resides in HBM, DRAM, or SSD, and a block not resident in HBM must be fetched before use.The decode step needs its required blocks in HBM; an unprefetched fetch stalls decode.Source: A. The tiered layer · p. 3

B. Placement, promotion, and eviction

Placement policies score blocks to determine which remain in faster tiers and which are demoted under capacity pressure. Prefetch lookahead moves anticipated blocks to HBM ahead of use, trading occupancy against stall avoidance.

B. Placement, promotion, and eviction

  • Recency, reuse frequency, and predicted reuse score blocks differently to determine tier placement and demotion under capacity pressure.Missed blocks are fetched and promoted; the lowest-scoring HBM block is demoted to DRAM, then SSD if needed.Source: B. Placement, promotion, and eviction · p. 3

C. Prefetch

  • Prefetch moves blocks expected in upcoming decode steps or a resuming session into HBM before they are needed.Lookahead depth trades HBM occupancy against stall avoidance.Source: C. Prefetch · p. 3

D. Cost model

The cost model weighs tiered capacity and cost against the fetch latency incurred on misses. The evaluation spans chat, agent, and document-QA workloads and tracks both capacity and serving outcomes.

D. Cost model

  • Tiered capacity is c_g + c_d + c_s, with blended cost dominated by cheaper tiers; misses add fetch latency that policy and prefetch must keep off the decode critical path.The model reports capacity and cost gains alongside measured stalls.Source: D. Cost model · p. 3

A. Workloads, baselines, and metrics

  • The evaluation uses long multi-turn chat, an agent loop revisiting recent context, and repeated document QA with heavy prefix reuse.Reported measures include tier hit rate, TTFT, TPOT and its stall component, sessions per GPU, throughput, and cost per session.Source: A. Workloads, baselines, and metrics · p. 3, Table I · p. 4
  • The placement layer assigns KV blocks across GPU HBM, CPU DRAM, and SSD, with promotion, demotion, eviction, and optional prefetch lookahead.The tiers represent small/fast, medium/warm, and large/cheap storage, respectively.Source: Figure 4 · p. 4

B. Policies and protocol

The study varies placement policies and prefetch settings across workloads, with a protocol designed to expose policy differences. Tiering’s sessions-per-GPU and cost gains track capacity, while placement policy affects migration and TTFT instead.

B. Policies and protocol

  • The policy sweep covers recency, reuse frequency, predicted reuse, a genuine EWMA next-touch predictor, block granularity, and prefetch lookahead.Baselines include GPU-only, full-CPU-offload, and prefix reuse.Source: B. Policies and protocol · p. 4
  • Chat uses six concurrent sessions of 4,000 accesses rather than one 24,000-access session, because the single-session run has no cross-request rereads for eviction policies to affect.Six sessions are the smallest concurrency at which the policies separate; the isolated-chat run yields byte-identical eviction results.Source: B. Policies and protocol · p. 4

V. RESULTS

  • The policy-by-workload grid reports PCIe migration traffic, and predicted_reuse is byte-identical to lru on every workload.At batch one, throughput is flat across policies, so migration bytes are the quantity placement controls.Source: Figure 5 · p. 5
  • At the 1:8:64 tier shape, tiering multiplies concurrent sessions per GPU and divides cost per session by the same 73× capacity factor, not by a placement-policy effect.The reported headline compares HBM-only with HBM+DRAM+SSD.Source: §V · pp. 4–5, Table III · p. 5

A. Sessions per GPU

Across the tier-ratio sweep, the sessions-per-GPU gain tracks added capacity rather than workload reuse. In the winning-policy hit split, DRAM absorbs most HBM overflow while SSD is a shallow backstop.

A. Sessions per GPU

  • The tiered-to-HBM-only sessions-per-GPU ratio stays between 72.68× and 73.22× for chat across the sweep, tracking the tier capacity multiple.The figure reports an approximately 73.02× ratio throughout the sweep, with similar behavior for other workloads.Source: Figure 6 · p. 5, A. Sessions per GPU · p. 5

B. Hit rate across tiers

  • Under the winning policy, chat serves 23.99% of accesses from HBM and 61.87% from DRAM, while agent and document QA serve 72.65% and 68.48% from DRAM.The passage identifies DRAM as the workhorse tier for these workloads.Source: B. Hit rate across tiers · p. 5
  • SSD serves 8.80% of document-QA accesses and 6.07% of agent accesses; the residual miss rate stays near 14.15% to 13.62%.SSD is described as a shallow backstop rather than a frequently hit tier.Source: B. Hit rate across tiers · p. 5

C. Which policy wins where

The migration-traffic winner depends on workload: recency favors chat, while reuse frequency favors agents and document QA. At batch one, compute dominates decode time, leaving migration traffic and TTFT as the policy-sensitive measures.

C. Which policy wins where

  • For chat, recency moves 27.38 GB versus 62.99 GB for reuse frequency, a 2.30× difference; reuse frequency wins for agents at 49.36 GB and document QA at 48.23 GB.The passage also reports reuse frequency as 1.60× and 1.63×, respectively, on those workloads.Source: C. Which policy wins where · pp. 5–6
  • The shipped predicted-reuse policy is byte-identical to recency; a genuine EWMA predictor ranks second on agent and document QA, behind reuse frequency.Predictive records 53.16 GB versus 49.36 GB for reuse frequency on agent, and 48.45 GB versus 48.23 GB on document QA.Source: C. Which policy wins where · p. 6

D. Where the latency goes

  • Decode compute dominates at roughly 13.2445 ms/token across policies, while fetch stall is under 1% of TPOT on every workload; TPOT poorly distinguishes policies, whose differences matter in TTFT and migration bandwidth.Source: D. Where the latency goes · p. 6
  • Tier-specific stall attribution is approximate because the simulator does not record which tier caused each stalled millisecond.The figure’s DRAM-versus-SSD stall split is attributed by hit share.Source: D. Where the latency goes · p. 6

E. Prefetch and the reuse structure

Prefetching fails to improve results on the tested bandwidth-charged link: even an oracle is slower than no prefetch in over half the cases. Tiered fetch stalls remain small relative to decode compute, and speculative transfers are not worthwhile when they compete with demand traffic.

E. Prefetch and the reuse structure

  • Across the grid, the causal predictor captures none of the oracle’s achievable gain, and the oracle is slower than no prefetch in over half the cases.On the contended link, speculative fetches delay demand fetches, and that cost exceeds the stall removed when reuse is not tightly predictable.Source: E. Prefetch and the reuse structure · p. 6
  • Tiered fetch stall is less than 1% of TPOT because compute dominates per-token decode latency.The DRAM/SSD share of stall is attributed approximately by hit rate.Source: Figure 8 · p. 7
  • Prefetch’s negative result is scoped to a link already carrying demand traffic, unlike systems that overlap transfers with abundant spare bandwidth.In this setting, speculative traffic is not worth its bandwidth cost.Source: E. Prefetch and the reuse structure · p. 7

F. The cost-latency frontier

At a fixed total budget, the cost-latency frontier is nearly vertical: composition changes cost far more than it changes P99 TTFT. With HBM and DRAM held fixed, deeper SSD increases sessions per GPU, but only document QA meets its GPU-only latency SLO in the tested range.

F. The cost-latency frontier

  • The fixed-budget sweep tests every HBM/DRAM/SSD composition summing to 192 blocks across three workloads and four policies.The capacity-under-SLO experiment instead fixes HBM at 32 blocks and DRAM at 64 while sweeping SSD depth, extending beyond the frontier’s total-budget cap.Source: F. The cost-latency frontier · p. 7, G. Capacity under a latency SLO: how… · p. 7
  • 8.38× cost range accompanies at most 10.1% of its own ceiling in P99 TTFT movement across the composition sweep.The cheapest frontier composition is 10/86/96 HBM/DRAM/SSD blocks, and the frontier leans toward the SSD-heavy end.Source: Figure 9 · p. 7, F. The cost-latency frontier · p. 7
  • The cost comparison assumes tier prices of $187.50, $3.50, and $0.08 per GB for HBM, DRAM, and SSD, respectively.These are default cost-model assumptions, not market quotes; the trade’s shape depends on their ratio.Source: F. The cost-latency frontier · p. 7

G. Capacity under a latency SLO: how deep can SSD go?

  • 10.77× more sessions per GPU at every workload comes from deepening SSD, but chat and agent never meet their SLO; document QA first meets it at 64 SSD blocks.At the deepest point, document QA matches its SLO with 0.000 ms headroom and still pays a mean stall of 4.42 ms.Source: G. Capacity under a latency SLO: how… · pp. 7–8, Figure 10 · p. 8

VI. DISCUSSION AND LIMITATIONS

Tiering delivers large capacity and cost gains independent of placement policy, while placement choices affect data movement in workload-specific ways. The authors qualify these findings by the synthetic workloads, batch-one simulator setting, and fixed cost assumptions.

VI. DISCUSSION AND LIMITATIONS

  • Synthetic generators favor the policy matching their reuse structure, so real-conversation reuse is lower and policy margins compress.The authors describe the synthetic separations as upper bounds on how cleanly policies divide.Source: §VI · p. 8
  • The simulator uses batch one on a single GPU, where decode is compute-bound and fetch stall is under one percent of TPOT.At larger batch or with faster decode kernels, the stall share and potential value of placement and prefetch would rise.Source: §VI · p. 8
  • Cost estimates use fixed tier prices, decode-stall attribution uses hit share, and quantization and eviction are held fixed to isolate placement.The authors say cost outputs are only as representative as the prices and that placement composes with quantization and eviction.Source: §VI · p. 8

VII. CONCLUSION

  • 73.02× more sessions per GPU and 62.04× lower cost per session come from tier capacity, independent of placement policy.The conclusion identifies recency for concurrent chat and reuse-frequency for repeated document QA as the placement choices that hold.Source: §VII · p. 8
  • Recency suits concurrent chat and reuse-frequency suits repeated document QA, while policies are identical for one isolated long chat.The discussion also reports recency's chat advantage and that reuse-frequency leads for agents and document QA; the shipped predicted-reuse policy is byte-identical to recency.Source: §VII · p. 8, §VI · p. 8

VI. DISCUSSION AND LIMITATIONS

  • The EWMA predictor still loses to reuse-frequency on the default agent cache, and prefetch costs more than it saves on a contended link.The discussion says prediction's value is confined to a narrow capacity window, and prefetch removes stalls only when spare bandwidth exists, which is largely absent at batch one.Source: §VII · p. 8, §VI · p. 8

Related papers

Loading 2609.16215v1…