PaperlayerView exact revision on arXiv

Source-linked AI summary

TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

Jay H. Park, Hyungjun Kim, Dong Kim

arXiv:2609.35065v1cs.DCcs.AIcs.LG

TL;DR

Reusable prefix KV can outgrow GPU memory, and a logical hit in SSD-backed memory may still need staging before retrieval. TempoKV times resource commitment by comparing runtime-estimated retrieval time with provider-estimated staging readiness. It substantially reduces protected-capacity cost versus immediate staging while retaining much of advance staging’s serving benefit.

  • Problem

    A logical KV hit in an SSD-backed memory hierarchy may not be ready for GPU retrieval, while staging on demand exposes SSD latency and immediate staging can reserve fast-tier capacity too early.

    Source: §1 · p. 1

  • Method

    TempoKV records reusable-KV hits as metadata-only claims and requests staging commitment when estimated time-to-use reaches estimated time-to-ready, subject to capacity checks.

    Source: §1 · p. 1

  • Results

    TempoKV reduces protected fast-tier byte-time per request by 63–91% versus immediate staging while retaining much of advance staging’s serving benefit; performance remains nearly unchanged as capacity falls from 100 to 25 GiB, and improves over LMCache-DAX on both workloads.

    Source: §1 · p. 2, §5.2 · p. 6, §5.3 · p. 6

  • Takeaways & Limitations

    Timing staging commitment retains much of advance staging’s serving benefit at substantially lower protected-capacity cost without changing request scheduling.

    Source: §6 · p. 6

Abstract

from arXiv · show

Reusable prefix key-value (KV) caches can outgrow GPU memory in large language model (LLM) serving. A memory-semantic flash hierarchy offers SSD-backed capacity with a limited fast tier, but a logical KV hit is not necessarily ready for GPU retrieval. Demand staging exposes SSD latency, whereas immediate staging can reserve fast-tier capacity long before retrieval begins. We present TempoKV, a timing-aware resource-commitment layer that separates early knowledge of reuse from the acquisition of staging resources. It records reusable-KV hits as metadata-only claims and requests commitment when the runtime-estimated time until retrieval falls to the storage-estimated time needed to make KV resident and protected against eviction. These estimates adapt to runtime progress and staging state, while commitment remains subject to available protected capacity. We implement TempoKV in vLLM and LMCache on an SSD-backed CXL memory device without changing request scheduling. Across two models and three prefix cache ratios, TempoKV reduces protected fast-tier byte-time per request by 63-91% versus immediate staging while retaining much of the serving benefit of advance staging. In a fast-tier capacity sweep, output throughput and p95 time to first token (TTFT) remain nearly unchanged as capacity decreases from 100 to 25 GiB. Compared with unmodified LMCache's Device-DAX L1 configuration, TempoKV reduces p95 TTFT by up to 48.0% and increases output throughput by up to 27.8%.

1 Introduction

Reusable prefix KV can be retained on SSD-backed memory, but a logical hit may still require staging before GPU retrieval. TempoKV times staging commitment using runtime and storage estimates, reducing protected-capacity cost while retaining much of advance staging’s serving benefit.

1 Introduction

  • A logical KV hit may still require SSD-to-fast-tier staging before the KV can be retrieved into GPU memory.The paper focuses on retained prefix KV between requests, not live KV accessed during generation.Source: §1 · p. 1
  • Demand staging exposes SSD latency, while commitment at hit discovery can reserve fast-tier capacity long before retrieval and constrain caching or other staging work.Queue rank indicates request order, not time until retrieval; the goal is to use lead time without holding protection unnecessarily early.Source: §1 · p. 1
  • TempoKV records reusable-KV hits as metadata-only claims and requests commitment when estimated time-to-use falls to estimated time-to-ready, subject to validity and capacity checks.Both timing estimates are reevaluated as runtime and staging state change.Source: §1 · p. 1
  • TempoKV extends vLLM and LMCache with an SSD-backed CXL provider while preserving request scheduling and existing prefix-lookup and GPU-transfer paths.The integration does not change device firmware or the CXL data path.Source: §1 · p. 1
  • TempoKV reduces protected fast-tier byte-time per request by 63–91% compared with immediate staging, while retaining much of the serving benefit of advance staging.Source: §1 · p. 2

2 Background and Motivation

Retained prefix KV can use SSD-backed memory because it remains quiescent until a later request matches it, but a logical hit does not guarantee fast-tier readiness. The motivation for TempoKV is to time commitment against retrieval rather than trigger it solely from early notice or queue rank.

2.1 Reusable KV on Memory-Semantic Flash

  • Retained prefix KV remains quiescent until a later request matches it, making it suitable for a high-capacity backing tier below HBM.Memory-semantic flash combines SSD-backed capacity with a smaller fast-memory tier; SSD-backed CXL is one realization, not a requirement for the timing problem.Source: §2.1 · p. 2

2.2 Memory Exposure Is Not Readiness

  • A logical hit does not establish fast-tier residency, and stage-ready means every required KV object is resident and protected against eviction.Staging commitment authorizes staging and reserves sufficient capacity to protect the matched KV through retrieval.Source: §2.2 · p. 2

2.3 The Staging-Timing Problem

  • Prior systems use queued-request prefetch or application signals, but advance notice alone does not provide a reliable wall-clock staging trigger.The paper distinguishes its timing decision from request scheduling and KV-read ordering.Source: §2.3 · p. 2
  • Median latency falls from 1.23 to 0.51 s as staging lead increases from 0 to 1 s, then plateaus; extending lead to 4 s raises protected byte-time from 8 to 32 GiB·s.The experiment uses an SSD-resident 64k-token prefix with an 8-GiB KV footprint.Source: §2.3 · p. 2
  • With identical rank-one triggers, timing margins range from 2.67 s early to 6.02 s late; adding backlog makes both target sizes late behind a 4-s predecessor.The tested factors include predecessor service time, KV footprint, and staging backlog.Source: §2.3 · p. 2
  • Timely commitment compares estimated time until retrieval with estimated time to stage-ready, subject to protected capacity, avoiding unnecessarily early reservation.The aim is readiness by expected retrieval without reserving fast-tier capacity earlier than necessary.Source: §2.3 · p. 2

3 TempoKV Design

TempoKV is a timing-aware commitment layer that leaves prefix matching and request scheduling unchanged. Its controller compares runtime-estimated time-to-use with provider-estimated time-to-ready, then attempts staging commitment subject to validity and protected-capacity constraints.

3 TempoKV Design

  • TempoKV combines a runtime adapter, controller, and storage-side provider, keeping runtime progress estimation separate from staging-state and fast-tier resource management.The runtime need not inspect storage internals, and the provider need not reconstruct request execution.Source: §3 · p. 3
  • Before commitment, a staging claim records the required KV objects and revisable time until retrieval without initiating I/O or reserving protected capacity.The overall design separates early knowledge of reuse from staging commitment.Source: §3 · p. 3

3.1 Two-Sided Timing

  • TTU projects running and queued work onto a shared runtime timeline using calibrated execution costs, resource releases, retrieval dependencies, and output-length estimates.The estimate is revised as runtime state changes; the projection assumes the target KV is stage-ready.Source: §3.1 · p. 3
  • TTR estimates when all required KV objects would become resident and protected, accounting for residency, queued or in-progress staging, and effective staging rate.The provider projects a hypothetical commitment; overlapping transfers are handled through aggregate service rate rather than summed transfer times.Source: §3.1 · p. 3

3.2 Timely Commitment and Replanning

  • A claim becomes eligible when TTU is at or below TTR or retrieval is requested, but commitment still requires provider validity and capacity checks.The controller reevaluates estimates and staging state; blocked claims do not prevent attempts for other eligible claims, and the timing interface allows estimator replacement.Source: §3.2 · pp. 3–4, §3.1 · p. 3
  • Protected staging shares the fast tier with ordinary caching, while an uncommitted retrieval lacking protection can use the ordinary demand path without reserving protected capacity.Demand retrieval preserves reuse but may incur SSD latency; the provider coalesces staging and reservations for shared KV objects.Source: §3.2 · p. 4
  • Claims progress through staging and readiness states, and protection lasts until GPU retrieval completes; releasing protection does not necessarily evict the KV.Cancellation of a committed claim retains protection until outstanding staging I/O and GPU accesses no longer need the objects.Source: §3.2 · p. 4

4 Implementation

TempoKV is integrated into vLLM and LMCache through a runtime adapter and a storage-side provider for SSD-backed CXL memory. The implementation preserves existing scheduling and retrieval paths while enforcing a protected-capacity budget.

4 Implementation

  • The vLLM adapter computes TTU after scheduling iterations, sends claim metadata to the LMCache service, queries readiness, and selects ordinary retrieval if protection is unavailable at retrieval time.The integration preserves vLLM scheduling and LMCache prefix-lookup and GPU-transfer paths.Source: §4 · p. 4
  • The provider uses an XCENA MX1P CXL Type-3 device, mapping KV manifests to device-memory ranges and using existing APIs for staging and protection.Device DRAM is the fast tier; the provider tracks staging and effective rate, and enforces the configured protection budget within the pin limit.Source: §4 · pp. 4–5
  • Figure 4 covers 50%, 75%, and 100% prefix cache ratios, with upper panels comparing performance against Demand and lower panels showing protected-capacity cost C.Higher p95 TTFT reduction and throughput gain are better; lower C is better.Source: Figure 4 · p. 5

5 Evaluation

The evaluation compares commitment policies, fast-tier capacities, and TempoKV against LMCache-DAX, measuring serving performance and protected-capacity cost. Across these experiments, timing-aware commitment reduces reservation cost while maintaining serving performance, including under capacity pressure.

5.1 Performance and Protected-Capacity Cost

  • The policy experiments vary prefix cache ratio across 50%, 75%, and 100%, comparing demand, immediate, queue-rank, static-TTR, and TempoKV commitment policies.The workload uses 12 NarrativeQA documents; TempoKV-Static fixes a model-specific staging rate at approximately 11 GiB/s and excludes queueing, contention, and ongoing staging progress.Source: §5.1 · p. 5

5.2 Sensitivity to Fast-Tier Capacity

  • With fast-tier capacity reduced from 100 to 25 GiB, TempoKV keeps throughput and p95 TTFT nearly unchanged at approximately 106 token/s and 5.9 s.At 25 GiB it has the highest throughput and lowest mean and p95 TTFT among six configurations, with lower C than Immediate and Queue-4; insufficient protection defers commitment for one request versus six for each of those policies.Source: Figure 5 · p. 6, §5.2 · p. 6

5.1 Performance and Protected-Capacity Cost

  • Across six model–ratio combinations, TempoKV reduces C by 63–91% versus Immediate; at 100%, it cuts p95 TTFT up to 25.7% and raises output throughput up to 20.4% over Demand.It also reduces C by 62–86% versus Queue-4; at 100%, it beats Queue-1 on both serving metrics at lower C in both models, while for Llama it trades 4.7% higher p95 TTFT and slightly lower throughput for 62% lower C versus Queue-4.Source: §5.1 · p. 6
  • For Llama at 100% prefix cache ratio, state-aware TTR estimation increases the p95 TTFT reduction over Demand from 14.5% to 20.5%, while C rises from 4.0 to 4.8 GiB·s/request.For Qwen at 75% and 100% ratios, TempoKV-Static already captures most serving gains.Source: §5.1 · p. 6

5.3 Comparison with the DAX-L1 Baseline

  • On NarrativeQA, TempoKV reduces p95 TTFT by up to 48.0% and increases output throughput by up to 27.8% over LMCache-DAX; LooGLE gains reach 32.4% and 15.5%, respectively.The comparison uses matched offered request rates on both workloads.Source: Figure 6 · p. 6, §5.3 · p. 6

6 Conclusion

TempoKV separates early knowledge of KV reuse from staging commitment in memory-semantic flash. It times commitment by comparing estimated time-to-use with estimated time-to-ready, retaining much of advance staging’s serving benefit at lower protected-capacity cost than immediate staging.

6 Conclusion

  • TempoKV compares runtime-estimated time-to-use with provider-estimated time-to-ready to time SSD-to-fast-tier staging commitment.Source: §6 · p. 6
  • TempoKV separates early knowledge of KV reuse from staging-resource commitment in memory-semantic flash hierarchies.Source: §6 · p. 6
  • On SSD-backed CXL memory, TempoKV retains much of advance staging’s serving benefit at substantially lower protected-capacity cost than immediate staging.Source: §6 · p. 6
Loading 2609.35065v1…