PaperlayerView exact revision on arXiv

Source-linked AI summary

Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving

Maharani A. P. Irawan, Kuntai Du, Jiayi Yao, Dongjoo Seo, Yuyang Huang, Haryadi S. Gunawi, Junchen Jiang

arXiv:2610.30791v1cs.LG

TL;DR

Production LLM workloads generate KV-cache footprints that exceed GPU and host-memory capacity, while SSD offloading faces an endurance challenge. Gandalf uses attention type and tier capacity to manage SSD admission, reporting 2–7.3× longer SSD lifetime than the best baseline across hybrid models and hardware capacities. The authors focus on SSDs and leave HDD integration for future work.

  • Problem

    Production LLM workloads generate KV-cache footprints beyond GPU and host-memory capacity, while SSD write endurance makes offloading at scale challenging.

    Source: §1 · p. 1

  • Method

    Gandalf uses attention type and tier utilization to place short-lived linear-attention states in DRAM and selectively admit higher-reuse full-attention caches to SSD.

    Source: §1 · p. 2

  • Results

    2–7.3× longer SSD lifetime than the best baseline across hybrid models and hardware capacities is reported, while preserving cache-hit rates.

    Source: §7 · p. 12

  • Takeaways & Limitations

    The authors state that Gandalf’s placement and endurance-management techniques also apply to other flash-based context-memory tiers, including HBF.

    Source: §2.2 · p. 3

    The evaluation focuses on commodity enterprise SSDs; integration with HDDs is left for future work.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

KV caching reduces the serving cost of large language models (LLMs), but the growing KV cache footprint is increasingly exceeds GPU and host-memory capacity. Therefore, people use SSD to expand the KV cache storage capacity with high I/O bandwidth. However, in production LLM workload, such as multi-round conversation and LLM agents, the total amount of KV cache writes significantly exceeds SSD capacity, shortening SSD lifetime from years to months. To improve SSD lifetime, a natural solution is to reduce the KV cache writes by gating the KV cache writes via historical access patterns. However, that significantly lowers the KV cache hit rate (and increase the LLM serving cost) since KV caches from different attention types might have identical historical access pattern but different future access patterns. Our insight is that, to reduce KV cache writes without hurting KV cache hit rate, one need to not only consider the access pattern, but also application-level information such as the attention type of KV cache. Leveraging this observation, we propose Gandalf , the first SSD KV cache offloading system with SSD life time awareness. Gandalf prioritize the admission of full-attention KV cache into SSD over the cache generated by other types of attention, which significantly increases SSD lifetime without sacrificing cache hit rate. Our evaluation shows that Gandalf extends the SSD lifetime by 7.3X on average, while having negligible impact on cache hit rate across multiple workload traces and wide range of hardware capacity. Gandalf design can be extended to other flash-based storage, such as High Bandwidth Flash (HBF).

1 Introduction

Production LLM workloads generate KV caches beyond GPU and host-memory capacity, while SSD write endurance makes aggressive cache admission costly. Gandalf uses attention type and tier capacity to reduce writes while preserving cache hits, reporting an average 7.3× SSD-lifetime increase with negligible hit-rate impact.

1 Introduction

  • Production LLM workloads generate KV caches beyond GPU and host-memory capacity, motivating SSD as a higher-capacity, comparatively cost-effective storage tier.Recomputing attention states for growing, reused contexts increases inference latency and cost.Source: §1 · p. 1
  • Existing SSD admission filters reduce writes but leave aggregate KV-cache write volume excessive for practical endurance.Frequency-based filtering gates admission after a cache demonstrates N× reuse.Source: §1 · p. 1
  • Full-attention KV caches are reused across future conversation rounds, whereas linear- and sparse-attention states are reused less frequently.This heterogeneity gives attention type potential as an SSD-placement signal.Source: §1 · pp. 1–2
  • Gandalf keeps longer-lived full-attention caches on SSD and short-lived linear-attention states in host memory to reduce writes without sacrificing hit rate.The approach uses attention-type-aware placement to focus SSD writes on caches with greater reuse potential.Source: §1 · p. 2
  • Gandalf’s capacity-aware policy filters SSD admission according to tier utilization and pressure to address mismatches between workload demand and fixed DRAM and SSD capacities.Underutilization of one tier can coexist with eviction pressure on the other, lowering cache-hit rate and potentially accelerating SSD write churn.Source: §1 · p. 2
  • 7.3× average SSD-lifetime extension with negligible cache-hit impact across a wide range of LLMs and hardware capacities is reported for Gandalf.The work also presents attention-type-aware placement as extendable to High Bandwidth Flash (HBF).Source: §1 · p. 2

2 Motivation

Growing, reused KV caches exceed practical memory capacity, while SSD writes under production workloads threaten device endurance. The motivation is to distinguish cache states by their future reuse so that writes with little cache-hit value can be avoided.

2.1 KV Caches Exceed Memory Capacity

  • 81–240 TB of KV-cache state per day and an 8.6TB peak working set in the analyzed traces exceed practical host-memory capacity, motivating storage beyond memory.The estimates come from two one-day SemiAnalysis Claude Code traces.Source: §2.1 · pp. 2–3

2.2 Why not HDD cluster?

  • At equal $20k TCO and aggregate I/O bandwidth, HDDs use more devices, power, and space yet have shorter projected lifetime when each KV-cache entry is reused at least 3×.The paper therefore focuses on SSDs for heavy-reuse multi-round conversation and agentic workloads, while leaving HDD integration for future work.Source: §2.2 · p. 3, Figure 2 · p. 3

2.3 KV Caching is Rapidly Killing SSDs

  • Even a 30-TB enterprise SSD can exhaust rated TBW in under a year under frontier-model and agentic workloads; write-through and write-back both persist states with little future reuse.Frequency-only filtering can still admit large, short-lived states without accounting for state size, layer-specific reuse, tier occupancy, or remaining write budget.Source: §2.3 · p. 3

2.4 How does prior work address SSD endurance?

  • Prior endurance approaches provision more write budget or selectively manage cache admission and eviction, but the surveyed cache policies do not explicitly optimize SSD endurance.Provisioning and retention-relaxation approaches increase available write budget rather than reducing host-level KV-cache writes.Source: §2.4 · pp. 3–4

2.5 SSD lifetime and Cache Hit Rate Trade-off

  • Storing more entries on SSD improves cache-hit rate and reduces TTFT but consumes finite endurance; storing fewer extends device lifetime while potentially lowering hit rate and increasing TTFT.With homogeneous cache entries, improving either objective comes at the expense of the other; breaking the trade-off requires identifying writes that contribute little to future reuse.Source: §2.5 · p. 4

2.6 Key Observation: Hybrid model has heterogeneous reuse pattern

  • Hybrid models have heterogeneous reuse: full-attention prefix KV caches are reused in every subsequent round, while sparse-attention states are typically used once before being superseded.Persisting short-lived states can consume endurance and capacity without adding a cache hit when the latest state can remain in DRAM, and may evict more reusable full-attention entries.Source: §2.6 · p. 4

3 Overview

Gandalf uses attention type as a reuse signal to place KV states across DRAM and SSD, then adapts admission and eviction to tier capacity and SSD write cost. An offline oracle motivates the policy, whose online implementation uses incremental candidate sets and heaps to reduce scoring overhead.

3.1 Attention type-aware placement

  • Attention type provides a placement signal because full-attention KV caches and linear-attention states differ in size and reuse, while prior reuse-boundary policies do not choose their storage tier.The strict policy retains linear states in DRAM and full-attention KV caches on SSD, evicting least-recently-used entries when a tier fills.Source: §3.1 · p. 5

3.2 Understanding the Potential

  • The attainable lifetime improvement rises with linear-state size relative to full-attention cache size and falls as more linear state must spill to SSD.The ideal keeps linear states in DRAM and full-attention KV on SSD, but requires enough DRAM for live linear states.Source: §3.2 · p. 5
  • Strict type-based placement can underuse tiers when workload state sizes exceed the preferred tier, so Gandalf adapts placement to capacity and expected reuse.Short sessions may need linear states to borrow SSD space, while long sessions may let full-attention KV borrow unused DRAM.Source: §3.2 · p. 5

3.3 Capacity-aware KV Cache Management

  • The capacity-aware policy approximates an oracle by choosing online whether to drop or move states, using expected reuse, tier occupancy, and SSD write and replacement costs.It retains the oracle’s objective of maximizing recomputation saved while accounting for write and eviction costs.Source: §3.3 · p. 6

3.3.1 Dropping a DRAM candidate.

  • Gandalf prices dropping a DRAM candidate by expected lost recomputation per byte and request of DRAM it frees, estimating next reuse from request history.For linear states, most are reused once or never; full-attention chunks use a history-based estimate of return probability.Source: §3.3.1 · p. 6

3.3.2 Moving a DRAM candidate to SSD.

  • Moving a DRAM candidate to SSD trades its endurance cost and any lost future hits from SSD victims against the recomputation avoided by retaining it.SSD write cost is scaled against a target lifetime write budget; while SSD has room, no replacement cost is charged.Source: §3.3.2 · p. 7

3.4 Optimization

  • Incremental candidate sets and lazy heaps limit rescoring to cache pressure events and heap heads, reducing decision work across millions of cached chunks.Candidates are restricted to leaves for dropping and root-side nodes for SSD moves; heaps organize drop, move, and SSD-victim choices.Source: §3.4 · p. 7

4 Implementation

The implementation validates a trace-based SSD-lifetime simulator against an SSD emulator, then uses the simulator to evaluate configurations before deploying the technique with LMCache and vLLM. Its lifetime estimates assume write-amplification factor 1.

4 Implementation

  • Validation connected vLLM and LMCache to a Claude Code trace, logged storage-facing operations, and replayed them on FEMU configured as a 1 TB, 0.3 DWPD, five-year SSD.The authors then deployed the confirmed technique in a production system using LMCache as a vLLM backend.Source: §4 · p. 7
  • The simulator estimates SSD lifetime from total bytes written per trace replay and the assumed daily replay volume, dividing SSD TBW by daily writes.The trace-based estimate was checked against lifetime observed by replaying workloads on an SSD emulator.Source: §4 · p. 7
  • The evaluation assumes an enterprise SSD rated at 1 DWPD for five years and varies SSD capacity.The simulator models transfers among SSD, CPU DRAM, and GPU HBM using specified hardware characteristics.Source: §4 · p. 7
  • The simulator assumes WAF=1 and omits SSD internal operations such as garbage collection and wear leveling, so it may overestimate lifetime for Gandalf and baselines.The authors state that reducing write volume can still reduce WAF and improve lifetime.Source: §4 · p. 7

5 Evaluation

The evaluation compares Gandalf with six cache-management baselines using real-world LLM workload traces and fixed DRAM with varying SSD capacity. It measures SSD lifetime, cache miss ratio, and cache load latency.

5.1 Setup

  • Six baselines are augmented with DRAM and SSD tiers while retaining their original admission and victim-selection rules.For baselines with hyperparameters, the evaluation sweeps configurations and shows the best on each trace.Source: §5.1 · pp. 7–8
  • Table 3 reports that the workload traces’ writes exceed the SSD’s workload DWPD rating.The evaluation uses Qwen Bailian Trace A, Claude Code WEKA SemiAnalysis, and AIPerf traces.Source: Table 3 · p. 8, §5.1 · p. 8
  • Experiments use Kimi-K3, with 72% linear-attention layers, 100GB of DRAM, and either a 1TB or 4TB SSD.The DRAM cache holds less than 1% of the peak active KV cache size for most traces.Source: §5.1 · p. 8
  • SSD lifetime measures years until the drive reaches its write-endurance limit; cache miss ratio and cache load latency are also evaluated.Cache load latency includes loading KV cache from DRAM or SSD, or recomputing absent entries.Source: §5.1 · p. 8

5.2 End-to-end Performance

With 100GB of DRAM and a 1TB SSD, Gandalf extends SSD lifetime while maintaining or reducing cache miss ratio across the evaluated traces. Its lifetime advantage varies by trace and by the selected miss-ratio operating point.

5.2 End-to-end Performance

  • 57.7 months at a 0.839 miss ratio on TraceA, versus 12.7 months for the longest-lived baseline at the same miss ratio.That baseline is HotPrefix with F=10; at α=0, Gandalf writes 2.0–6.5× fewer bytes than the lowest-miss baselines across traces.Source: §5.2 · p. 8
  • 7.3× average SSD-lifetime improvement while maintaining or reducing cache miss ratio, compared with the baselines.At its lowest miss ratio, Gandalf extends lifetime over the lowest-miss baseline by 2× on TraceA, 3.3× on ClaudeCode, and 6.5× on AIPerf.Source: §5.2 · p. 8
  • On ClaudeCode, Marconi with α=0 and LRU have slightly lower miss ratios than Gandalf—0.045 and 0.047 versus 0.052—but write 3.3× more to SSD.The reported lowest-miss comparison therefore includes a trace-specific miss-ratio trade-off.Source: §5.2 · p. 8

5.3 Cache Ratio Sensitivity

Gandalf’s results persist across tier-capacity ratios and hybrid-model attention-state mixes. Its write-fee multiplier α lets operators trade cache misses for SSD lifetime.

5.3 Cache Ratio Sensitivity

  • With total capacity fixed at 4TB, Gandalf’s miss ratio falls from 0.75 to 0.6 as DRAM grows, while SSD lifetime stays between 29 and 35 months.HotPrefix reaches 33–122 months but has a 0.852–0.856 miss ratio, above the trace’s maximum achievable 0.6.Source: §5.3 · p. 8
  • 5–25% lower cache miss ratio than the best baseline at every tested capacity ratio, with SSD lifetime extended by up to 120× in tested 4TB-capacity scenarios.Figure 8 varies DRAM/SSD capacity while holding their combined capacity at 4TB.Source: §5.3 · pp. 8–9, Figure 8 · p. 9

5.4 Trade-Off Hyperparameter Sensitivity

  • From α=1 to α=64, SSD lifetime grows 6.5× on TraceA, 2.3× on ClaudeCode, and 2.7× on AIPerf, while miss ratio rises by 0.016, 0.071, and 0.172, respectively.Gandalf’s default α=1 balances lifetime and cache hit ratio across traces; larger α makes SSD admission more selective.Source: §5.4 · p. 9
  • Operators can tune α to meet their desired trade-off between SSD lifetime and cache hit ratio, and Gandalf is robust across a wide range of α values.Figure 7 plots SSD lifetime against cache miss ratio across traces and hyperparameter settings.Source: §5.4 · p. 9, Figure 7 · p. 9

5.5 Model Linear-to-Full Sensitivity

Gandalf’s lifetime advantage holds across hybrid models with different linear-to-full attention cache-size ratios. Its results also reflect selective writes and tier placement based on attention type.

5.5 Model Linear-to-Full Sensitivity

  • 3.6–7.1× longer SSD lifetime than HotPrefix and 3.6–6.6× longer than LRU-K across four hybrid models.Gandalf reduces miss ratio relative to both baselines on three models; on Nemotron-H, its miss ratio is similar to LRU-K.Source: §5.5 · pp. 9–10
  • The lifetime gain over LRU-K grows from 3.6× to 5.8× and 6.6× among Nemotron-H, Gemma3-27B, and Kimi-K3 as the state-to-KV ratio increases.Even Qwen3.5-27B, the model with the smallest ratio, shows a 3.9× gain.Source: §5.5 · p. 10

5.6 Where does the improvement come from?

  • Across time, a higher fraction of Gandalf’s SSD writes are hit at least once than LRU-K’s, and Gandalf maintains a high cache hit rate as SSD fills.Figure 10 shows the write-hit fraction over time; Gandalf avoids many writes for linear states that are not reused on SSD.Source: §5.6 · p. 10, Figure 10 · p. 10
  • Gandalf writes 8.8× fewer linear states than LRU-K at K=1, and 57% of Gandalf’s written linear states are hit on SSD.LRU-K at K=1 writes 120.6k linear states, of which only 21% are ever hit on SSD; with K=2, only 13% are hit there.Source: §5.6 · p. 10
  • Gandalf writes 33% fewer full-attention KV chunks than LRU-K at K=1, serves 18% more SSD hits, and extends SSD lifetime 1.7× at a similar miss ratio.It writes 92TB rather than 153TB, with lifetime of 98 versus 59 days and miss ratio of 0.099 versus 0.098.Source: §5.6 · p. 10
  • Gandalf serves 87% of full-attention hits from SSD and 90% of linear-state hits from DRAM, compared with 74% and 67% for LRU-K.Figure 11 shows the tier serving cache hits by attention type; Gandalf’s placement uses attention type as a proxy for reusability.Source: §5.6 · p. 10, Figure 11 · p. 10

5.7 Comparing Gandalf to each baseline.

Gandalf reduces SSD writes and extends lifetime relative to write-back and admission-filtering baselines, with a miss-ratio trade-off that depends on its α setting.

5.7 Comparing Gandalf to each baseline.

  • 73× fewer bytes than LRU (0.58 against 42.2TB) and 73× longer SSD lifetime (253 against 3.5 days); Figure 12 attributes the gain to fewer writes and a higher Read/Write ratio.Gandalf writes 3.2× fewer full-attention bytes than LRU (0.53 against 1.70TB); Marconi writes 42.7TB and lasts 3.4 days, while Workload-aware writes 63.5TB and lasts 2.3 days.Source: Figure 12 · p. 11, §5.7 · p. 11
  • HotPrefix at F=10 writes 0.38TB and lasts 388 days, but misses 0.839 of chunks; Gandalf at α=1 lasts 7.6× longer than HotPrefix at F=1 with a 0.016 higher miss ratio.At F=1, HotPrefix writes 4.4TB and lasts 33 days, but 51% of its written linear states are never requested again.Source: §5.7 · p. 11
  • LRU’s SSD lasts 21.5 days; at α=1, Gandalf lasts 11.8× longer with a 0.068 higher miss ratio, and on TraceA its miss ratio is 0.818 versus LRU’s 0.697.Gandalf’s SSD lasts 73× longer than LRU’s; a smaller α recovers these hits at a shorter lifetime.Source: §5.7 · p. 11

5.8 How much is the cache load latency affected?

Gandalf trades slightly higher cache-load latency for longer SSD life, and its lower SSD wear can reduce total request cost. Testing more move candidates does not improve cache misses.

5.8 How much is the cache load latency affected?

  • Gandalf at α=1 lasts 4.2× longer than LRU (17.0 against 4 months) at 0.78s cache-load latency, 0.12s higher than the 0.66s baselines.At similar latency, Gandalf lasts 3.7× longer than LRU-K at K=2; at α=64, it lasts 5.3× longer than LRU-K at K=3 while loading faster.Source: §5.8 · p. 11
  • $0.30 per 1k requests, Gandalf at α=1 costs 2.5× less than the cheapest baseline, LRU-K at K=2 ($0.74 per 1k requests).Figure 14 compares average recomputation-plus-SSD-wear cost; Gandalf's SSD wear is 3.0× lower than any baseline ($0.16 against $0.49–0.73).Source: §5.8 · p. 11, Figure 14 · p. 12

5.9 The optimization does not affect the performance negatively.

  • Checking up to 64 move candidates per slot size leaves miss ratios unchanged at 0.823 on TraceA and 0.036 on AIPerf, while TraceA writes rise 1.4%.On AIPerf, the first candidate is never blocked, so the result is identical; the single-candidate policy therefore loses no hits.Source: §5.9 · p. 12

6 Related Work

Prior work addresses tier placement, reuse prediction, and flash wear through different mechanisms. Gandalf's closest related work uses inferred agent-segment lifetimes for HBM and high-bandwidth flash placement, whereas Gandalf targets DRAM and SSD using hybrid-attention semantics.

6 Related Work

  • Tiered KV-cache systems extend GPU memory across DRAM, SSD, or distributed storage [7] [13], while selective fetching and GPU-directed SSD I/O reduce lower-tier access overhead [20] [37].These systems primarily optimize capacity, latency, or throughput rather than SSD endurance.Source: §6 · p. 12
  • Prior admission, eviction, and compression methods use reuse signals or reduce KV footprint [12] [33] [34] [44] [46] [48], but most do not jointly manage DRAM placement, SSD admission, and SSD eviction using attention-type reuse heterogeneity and explicit endurance cost.MeritKV, Bidaw, LRU-K, and NVIDIA KVBM incorporate reuse utility, storage cost, or write awareness [35].Source: §6 · p. 12
  • Flash-cache filtering and device-level techniques control wear or reduce physical writes, while LLM-serving systems optimize flash data paths, writeback, and reclamation [45] [54].These mechanisms are complementary to Gandalf because they do not use attention semantics to value individual KV states.Source: §6 · p. 12
  • Lachesis places agent KV segments across HBM and high-bandwidth flash using lifetimes inferred from the agent harness [50], whereas Gandalf targets host DRAM and SSD without harness-specific information.Gandalf jointly considers capacity, recomputation benefit, and SSD write cost.Source: §6 · p. 12

7 Conclusion

Gandalf uses attention type to estimate KV-cache reuse and balances expected hits against SSD wear when choosing placement. Across hybrid models and hardware capacities, it improves SSD lifetime by 2–7.3× over the best baseline.

7 Conclusion

  • Gandalf retains short-lived linear-attention states in DRAM and offloads full-attention KV caches with high reuse potential to SSD, balancing expected hits against device wear.Its placement policy filters writes based on reuse potential and wear cost while preserving cache hit rates.Source: §7 · p. 12
  • Across a range of hybrid models and hardware capacities, Gandalf improves SSD lifetime by 2–7.3× over the best baseline.Source: §7 · p. 12
  • The conclusion identifies extension to other flash-based storage as future work and calls for considering write wear alongside SSD-offloading performance benefits.Source: §7 · p. 12

Related papers

Loading 2610.30791v1…