PaperlayerView exact revision on arXiv

Source-linked AI summary

Characterizing Predictability-Latency Trade-offs of KV-Cache SSD Offloading in LMCache for LLM Serving Systems

Ying He, Dingsen Shi, Yanbo Dai, Donger Chen, Harry Fu, Chris Tsu, Wang Feng, Song Fu

arXiv:2610.21249v1cs.OS

TL;DR

The paper examines how LMCache’s SSD offload behaves under two realizable storage configurations, addressing limited block-device-level characterization. It finds a trade-off: page-cache warmth can hide legacy reads, while O_DIRECT makes retrieval cost explicit and stable, with a latency penalty in an exact-repeat workload.

  • Problem

    LMCache operators choose between buffered I/O with the kernel page cache and direct I/O, but the performance consequences of that storage-path choice remain undercharacterized.

    Source: §I · p. 1

  • Method

    The paper characterizes block-layer behavior across four stateless workloads and compares LMCache 0.3.0 buffered I/O with LMCache 0.3.16 prefix-aware deduplication and O_DIRECT.

    Source: §I · p. 1

  • Results

    Legacy cold-cache reads reach ≈1 GB in 683 ms; in Set A, writes fall from 1.62 GB to 11.75 MB while warm TTFT rises from 27 ms to 93 ms.

    Source: B. An eviction experiment reveals the… · p. 3, A. Set A becomes predictable and slower · p. 3, §VII · p. 6

  • Takeaways & Limitations

    For current LMCache SSD offload, the central design question is whether deployment values predictable latency more than average warm-path speed.

    Source: §VII · p. 6

    The comparison is observational across two deployment snapshots, not a causal ablation or a full factorial comparison.

    Source: A. Two deployment snapshots · p. 2

Abstract

from arXiv · show

KV-cache offload is widely used to stretch GPU memory for LLM serving, but its storage behavior has not been characterized at the block-device level. In this paper, we study LMCache through realworld multi-session workloads that span same/different context x same/different prompt, using over 100 stateless requests per workload. Our comparison is observational rather than factorial: it contrasts two realizable deployment snapshots -- LMCache 0.3.0 with buffered I/O and the kernel page cache, and LMCache 0.3.16 with prefix-aware deduplication and O_DIRECT. Key findings define the paper. First, the legacy stack appears almost read-free at the SSD layer in all four workloads, but a flush_ram experiment shows that this is conditional on page-cache warmth: once the cache is evicted, the same path issues about 1 GB of sequential reads in 683 ms. Second, deduplication delivers a dramatic write reduction only for the exact-repeat workload: one set collapses from 1.62 GB to 11.75 MB of writes, while the other three sets remain in the 2-5 GB range. Third, O_DIRECT converts a hidden and bimodal read cost into an explicit and stable one: on one set, warm TTFT rises from 27 ms to 93 ms because each of the 99 warm requests re-reads the same ~12 MB partial chunk from SSD. The main conclusion is therefore not "newer is better," but a system trade-off: predictability versus average-case latency.

I. Introduction

The paper characterizes LMCache SSD behavior across four workload patterns and contrasts two realizable storage-path snapshots. Its central finding is a trade-off between warm-path speed and predictable storage cost, not a simple upgrade from one version to another.

I. Introduction

  • LMCache 0.3.0 uses buffered I/O with the kernel page cache, whereas LMCache 0.3.16 adds prefix-aware deduplication and supports O_DIRECT.The paper treats these as two realizable deployment points rather than claiming one version dominates.Source: §I · p. 1
  • Buffered I/O and the page cache can make SSD reads disappear in warm-cache conditions, but that benefit depends on host memory state outside LMCache’s control.O_DIRECT makes reads explicit and stable but pays that cost on every retrieval, even when data might otherwise remain hot in RAM.Source: §I · p. 1
  • The study combines four stateless workload patterns with page-cache eviction to show that LMCache SSD offload trades average warm-path speed for predictability.The workloads span exact repetition, fixed-context/varying-prompt traffic, fixed-prompt/varying-context traffic, and fully varied traffic.Source: §I · p. 1
  • Prefix-aware deduplication delivers its strongest write reduction only for exact-repeat traffic; varied workloads still write GB-scale data.The paper’s block-layer characterization and eviction experiment show why the apparent zero-read behavior of the legacy path is conditional.Source: §I · p. 1

II. Relation to Prior Work

Prior work addresses LLM serving, KV-cache reduction and reuse, and coordination between computation and loading. This paper focuses instead on how buffered and direct storage paths differ at the SSD layer.

II. Relation to Prior Work

  • LLM serving research optimizes batching, memory management, and phase placement, but typically treats offloaded KV access as an abstract tier.The paper characterizes the differing SSD-layer behavior of buffered and direct storage paths.Source: §II · p. 2
  • KV-cache reduction and reuse methods can save compute, but prior work rarely measures what the storage device sees when reuse occurs.The cited approaches include pruning, compression, quantization, selective retention, and cross-request sharing.Source: §II · p. 2
  • Compute-versus-load work co-optimizes prefill computation and KV loading, while this study links that choice to block-level storage behavior.In short-context partial-chunk regimes, loading reused KV can cost more than recomputing the tail it is meant to save.Source: §II · p. 2
  • The paper characterizes a practical open-source KV-cache layer rather than proposing a new serving engine or KV policy.Its distinction is the shift from hidden read cost to explicit read cost.Source: §II · p. 2

A. Two deployment snapshots

The evaluation compares two archived deployment snapshots and traces I/O at the block layer. Its workloads are multi-session, stateless, and confined to short-context partial-chunk regimes.

A. Two deployment snapshots

  • The comparison is observational: LMCache 0.3.0 and 0.3.16 traces come from different dates, vLLM versions, and SSDs, rather than a full factorial design.The new buffered-I/O path with save_unfull_chunk=True was unstable in the environment and appeared to trigger a use-after-free crash.Source: A. Two deployment snapshots · p. 2

B. Tracing and workloads

  • bpftrace records block-layer I/O submission rather than completion, so short submit-rate microbursts are not physical-throughput measurements.The cited Set A burst implies about 10.4 GB/s, above the stated PCIe Gen4 x4 ceiling of about 8 GB/s.Source: B. Tracing and workloads · p. 2
  • Each workload runs 100 independent stateless requests, and all four remain below two full 256-token chunks.The sets represent exact-repeat behavior, fixed context with varying queries, fixed questions with changing snippets, and an unconstrained baseline; they stress incomplete tails.Source: B. Tracing and workloads · p. 2, Table I · p. 2

A. Legacy LMCache looks almost read-free

At the block layer, the legacy stack appears nearly read-free across the four workloads, while writes remain substantial and vary by workload. Buffered I/O also separates observed SSD write timing from LMCache’s logical persistence decisions.

A. Legacy LMCache looks almost read-free

  • Legacy block-layer reads are only 0–0.05 MB across the four workloads, while writes range from 1.62 GB in Set A to 3.79 GB in Set B.The pattern is consistent with write-through behavior without prefix-aware deduplication, even for repeated content.Source: A. Legacy LMCache looks almost read-free · p. 2
  • Buffered writes arrive in irregular kernel-writeback spikes rather than 100 request-aligned bursts, obscuring when LMCache chose to persist chunks.The few legacy reads are metadata activity, not LMCache data fetches; metadata-read-to-data-write ratios remain near 1:900–1:1000.Source: A. Legacy LMCache looks almost read-free · pp. 2–3

B. An eviction experiment reveals the hidden cost

Evicting the page cache exposes the SSD reads hidden by LMCache 0.3.0’s warm-cache behavior. The cold path reads about 1 GB sequentially in 683 ms.

B. An eviction experiment reveals the hidden cost

  • The experiment populated roughly 1 GB of LMCache data, forced memory pressure and cache eviction, then re-read the same file while tracing the block layer.Cache eviction used drop_caches plus explicit RAM pressure.Source: B. An eviction experiment reveals the… · p. 3
  • ≈1 GB of reads in 683 ms: after cache eviction, the block layer records a dense sequential scan at about 1.43 GB/s.The trace contains 8,002 read-ahead events of 128 KB each.Source: B. An eviction experiment reveals the… · p. 3
  • The legacy stack’s apparent zero-read behavior is conditional on RAM residency, producing a warm-cache/cold-cache bimodal cost model.With a cold page cache, reads become storage-bound.Source: B. An eviction experiment reveals the… · p. 3

A. Set A becomes predictable and slower

On Set A, O_DIRECT and prefix-aware deduplication sharply reduce writes but make SSD retrieval explicit on every warm request. The warm path is slower yet stable.

A. Set A becomes predictable and slower

  • 1.62 GB to 11.75 MB: Set A writes fall by about 138× under LMCache 0.3.16 + O_DIRECT as identical requests reuse the persisted chunk.Only the first request persists the chunk.Source: A. Set A becomes predictable and slower · p. 3
  • 9,405 reads and 1.16 GB of SSD traffic: the 99 warm Set A requests each trigger 95 reads and transfer 11.75 MB.The repeated retrieval makes the deduplication write reduction come with an explicit read cost.Source: A. Set A becomes predictable and slower · p. 3
  • 93.1 ms versus 27 ms mean warm TTFT: the new stack’s Set A path is slower, while its 99 warm requests mostly remain within 89–105 ms.The trace shows only a few lower outliers early in the run.Source: A. Set A becomes predictable and slower · p. 3

B. Why the read cost is high: partial-chunk amplification

Set A’s short unmatched tail still requires loading a full LMCache chunk from SSD. Across the varied workloads, chunk-level deduplication usually fails to prevent GB-scale writes.

B. Why the read cost is high: partial-chunk amplification

  • About 6.7× read amplification: Set A’s 14-token unmatched tail requires reading the full ∼12 MB chunk because GPU prefix blocks and LMCache chunk files have different granularities.vLLM uses 16-token GPU prefix blocks, while LMCache persists 256-token chunks; the roughly 94-token request leaves a 14-token tail after five full prefix blocks match.Source: B. Why the read cost is high:… · p. 3

C. Set B/C/D show dedup gains are narrow

  • 3.97 GB, 2.03 GB, and 4.83 GB: Sets B, C, and D write these totals, respectively, under the new stack.Their varied prompts or contexts change chunk content, leaving little to retrieve and almost no LMCache hit tokens.Source: C. Set B/C/D show dedup gains are narrow · p. 3
  • Set B rewrites its shared short context because prompt variation changes the full request’s chunk hash, yielding 3.97 GB of writes and no reads.The trace averages about 39.7 MB written per request.Source: C. Set B/C/D show dedup gains are narrow · p. 3
  • Set C writes 2.03 GB and Set D writes 4.83 GB; Set D also has no reads and 59.2 ms warm TTFT.Set C’s single 128 KB read across the run does not represent meaningful KV reuse.Source: C. Set B/C/D show dedup gains are narrow · p. 3
  • Deduplication’s strongest write collapse depends on repeated content surviving chunk boundaries unchanged; that condition fails across all three varied workloads.The largest benefit is therefore a best-case benefit, not a broad one.Source: C. Set B/C/D show dedup gains are narrow · pp. 3–4

A. The real choice is predictability versus average throughput

On Set A, the legacy stack is fast when the page cache is warm, while the newer stack makes retrieval explicit and consistent. The result is a trade-off between average-case latency and predictability.

A. The real choice is predictability versus average throughput

  • 27 ms warm TTFT: Set A stays near this level with the legacy stack when host RAM keeps the working set resident.That benefit depends on memory conditions outside LMCache’s control.Source: A. The real choice is predictability… · p. 4
  • About 95 submitted reads, 11.75 MB transferred, and 93 ms TTFT: every Set A warm request behaves similarly under the new stack.The explicit read path removes the dependency on whether the kernel still caches the chunk.Source: A. The real choice is predictability… · p. 4
  • The newer path is worse on average in this single-user short-context setting, but more predictable for operators focused on tail latency or service-level objectives.The comparison is a cost-model distinction, not a version ranking: buffered I/O hides cost opportunistically, while direct I/O exposes it deterministically.Source: A. The real choice is predictability… · p. 4

B. Implications for KV-cache design

The value of KV reuse depends on the size and storage granularity of the reused object. The implications point toward finer-grained persistence and policies that weigh loading against recomputation.

B. Implications for KV-cache design

  • After page-cache eviction, replaying the data path generates 8,002 128 KB reads and transfers ≈1 GB in 683 ms.The experiment shows the read traffic hidden by a warm page cache.Source: Figure 2 · p. 5
  • Under O_DIRECT, Set A warm TTFT stays mostly within 89–105 ms after the first cold request.The figure presents this as low variance with a higher latency floor.Source: Figure 3 · p. 5
  • If the unmatched suffix is tiny, retrieving a whole partial chunk may cost more than recomputing it.The paper suggests materializing only the unmatched suffix to preserve deduplication benefits without the full-chunk read cost.Source: B. Implications for KV-cache design · p. 5
  • Separating shared prefixes from varying tails could improve deduplication for stateless traffic, with estimated write savings rising from ∼15% for Set B to ∼80% for aligned 1024-token RAG prefixes.The expected gain scales with how much of a request is shared.Source: B. Implications for KV-cache design · p. 5
  • Hybrid compute-or-load policies are relevant because loading and recomputing should be co-optimized.The measurements provide a block-level explanation for this policy direction, including on local SSDs.Source: B. Implications for KV-cache design · p. 5

C. Operational takeaways

The preferred LMCache storage path depends on repetition structure, host-memory slack, and the desired latency stability. Cache-hit rate alone is insufficient without considering the size of the retrieved object.

C. Operational takeaways

  • Buffered I/O suits short repeated prompts when host RAM is abundant and interference is low; O_DIRECT suits operators prioritizing stable storage budgets and warm latency.The legacy Set A trace has essentially no SSD data reads and low warm TTFT; O_DIRECT exposes retrieval cost independently of host memory pressure.Source: C. Operational takeaways · p. 5
  • A cache hit can underperform when a coarse storage object takes longer to load than the compute it would save.Interpret hit rate jointly with retrieved object size; Set A hits expensively, while Sets B/C/D mostly miss yet have lower warm TTFT.Source: C. Operational takeaways · p. 5
  • Exact repetition shifts Set A from always writing to write-once/read-many, while varied traffic remains write-dominated under both stacks.The crossover may shift with longer contexts, larger shared prefixes, and chunk-aligned RAG prompts; the results support treating SSD-backed KV reuse as workload-specific.Source: C. Operational takeaways · p. 5

VII. Conclusion

The conclusion frames LMCache SSD offload as a systems trade-off, not an upgrade story. Buffered I/O hides reads only while the page cache is warm, whereas O_DIRECT makes retrieval explicit and stable but raises Set A warm TTFT; deduplication’s largest write benefit is limited to exact repeats.

VII. Conclusion

  • On Set A, O_DIRECT raises warm TTFT from 27 ms to 93 ms, while its largest deduplication write benefit is confined to exact-repeat traffic.The paper’s practical conclusion is to weigh predictable latency against average warm-path speed; resolving buffered-path instability in the newer version is identified as the highest-leverage next step.Source: §VII · p. 6

Related papers

Loading 2610.21249v1…