PaperlayerView exact revision on arXiv

Source-linked AI summary

SARIA: Persistence-Aware KV Caching with Coordinated DRAM--SSD Reload for LLM Agents

Hui Li, Qihui Zhou, Xiao Yan, Shumeng Zhang, Zijie Dai, Zequn Gong, Mingcong Song, Guoping Long, Hongjie Si, James Cheng

arXiv:2610.31169v1cs.OS

TL;DR

Long-prefix agent workloads make KV caching useful, but SSD caching faces endurance and reload-bandwidth constraints. SARIA uses reuse-aware admission and coordinated DRAM–SSD reloads; it reduces SSD writes relative to full duplicate caching and mean TTFT relative to DRAM-only caching.

  • Problem

    SSD caching for agent KV state is constrained by finite write endurance and limited SSD parallelism for reloads.

    Source: §1 · p. 2

  • Method

    SARIA uses session reuse and eviction history for selective SSD admission, then combines partial DRAM promotion with coordinated GDS and DRAM reloads.

    Source: §1 · p. 2, §4.4 · p. 7, §4.5 · p. 7

  • Results

    57.1–58.9% fewer SSD writes than full duplicate caching and 42.3–79.7% lower mean TTFT than DRAM-only caching.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across TraceLab-derived workloads and full-attention and hybrid-attention models, SARIA reduces SSD writes while reducing mean TTFT.

    Source: §7 · p. 12

    When GDS alone saturates GPU ingress bandwidth, coordination offers no benefit and its overhead may reduce performance.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Multi-turn LLM agents usually reuse long context prefixes, making key value (KV) caching crucial for reducing prefill latency. SSDs provide a low-cost extension to limited DRAM capacity but introduce two key challenges. Eagerly persisting all KV cache blocks incurs substantial write traffic and rapidly consumes finite SSD endurance. Meanwhile, the small number of SSDs provisioned in typical GPU servers limits aggregate reload bandwidth, making SSD-resident KV cache much slower to retrieve than that in DRAM and inflating prefill latency. We present \textsc{Saria} (Selective Admission and Reload for Inference Acceleration), a KV caching system that jointly manages DRAM and SSD by exploiting the session-level temporal locality of LLM agents to reduce SSD writes and accelerate cache reload. On the write path, \textsc{Saria} uses eviction and reuse history to selectively persist KV cache blocks that exhibit reuse opportunities beyond their DRAM residency, avoiding unnecessary writes for KV cache already hosted in DRAM. On the read path, \textsc{Saria} promotes selected SSD-hit KV cache blocks from GPU memory into DRAM after prefill, allowing subsequent reuse within the same session to reduce SSD read traffic while limiting DRAM contention. A transfer coordinator paces DRAM reads while SSD transfers are active, allowing both paths to make progress and efficiently share GPU ingress bandwidth. We implement \textsc{Saria} on LMCache MP and evaluate it using workloads from TraceLab and WEKA, covering both full-attention and hybrid-attention models. The results show that \textsc{Saria} reduces SSD writes by 57.1--58.9\% compared with popular full duplicate caching and the mean time to first token (TTFT) by 42.3--79.7\% over DRAM-only caching.

1 Introduction

Agent workloads reuse long context prefixes, but limited DRAM capacity makes retaining all reusable KV cache difficult. SARIA uses reuse history to reduce unnecessary SSD writes and coordinates DRAM–SSD reloads; its evaluation reports lower write volume and TTFT against the stated baselines.

1 Introduction

  • Eagerly persisting every newly generated KV block creates substantial write traffic that consumes finite SSD endurance.At the reported GLM-5.2 prefill rate, a 4-TB, 1-DWPD SSD would exhaust its five-year write budget in about 70 days of continuous operation.Source: §1 · pp. 1–2
  • Typical GPU servers have too few SSDs to match GPU ingress bandwidth, making SSD-resident KV reloads a bottleneck.Adding drives consumes bays and PCIe resources and increases cost.Source: §1 · p. 2
  • SARIA uses DRAM eviction and reuse history to persist blocks likely to be reused after eviction, while allowing thresholds to trade SSD writes against cache coverage.It also persists exceptionally popular blocks before eviction.Source: §1 · p. 2
  • After prefill, SARIA promotes selected SSD-hit blocks into DRAM and paces DRAM transfers during GDS reloads so both sources share GPU ingress bandwidth.Subsequent requests can reload different KV portions concurrently from DRAM and SSD.Source: §1 · p. 2
  • 57.1–58.9% fewer SSD writes than full duplicate caching and 42.3–79.7% lower mean TTFT than DRAM-only caching.The evaluation covers TraceLab and WEKA workloads, full-attention and hybrid models, and single- and dual-SSD configurations.Source: §1 · p. 2

2 Background and Challenges

KV caching spans GPU memory, DRAM, and SSD, with different reuse requirements for full-attention and hybrid models. SSD tiers extend capacity, but endurance limits writes and insufficient drive parallelism constrains reload bandwidth.

2.1 Hierarchical KV Caching

  • Full-attention KV can be reused at matched prefix boundaries, while hybrid models with recurrent layers also need checkpoints at those boundaries.SSD caching extends the hierarchy beyond GPU memory and DRAM; reloads can stage through DRAM or use direct I/O such as GDS.Source: §2.1 · p. 3

2.2 Endurance-Constrained SSD Provisioning

  • At 1 DWPD, sustaining B GB/s per GPU for five years requires at least C_min = 86.4B TB of aggregate SSD capacity per GPU.Under full duplicate caching, endurance can require substantially more SSD capacity than effective caching does.Source: §2.2 · p. 3

2.3 Limited SSD Reload Bandwidth

  • At the same PCIe generation, at least four typical x4 NVMe SSDs must operate in parallel to match one x16 GPU link.Dense GPU servers commonly provide fewer drives than needed to match aggregate GPU ingress bandwidth, and adding drives consumes resources and increases cost.Source: §2.3 · p. 3

3 Observations and Motivations

The observations examine how session reuse interacts with DRAM pressure and SSD persistence. They show that reuse often remains in DRAM at lower pressure, while aggressive promotion can increase DRAM competition and worsen tail transfer times.

3 Observations and Motivations

  • The TraceLab analysis preserves each session’s request sequence and inter-request intervals while varying session-start windows to create different cache-pressure levels.It replays 350 reconstructed sessions, averaging 18.4 requests per session and 83.7K prompt tokens per request.Source: §3 · p. 4

3.1 KV Reuse under Cache Pressure

  • In the 12-hour setting, 79.3% of cache-hit requests obtain all cached blocks from DRAM, which supplies 76.9% of cache-read bytes; in the 2-hour setting, the figures fall to 52.4% and 44.2%.Under lower load, 82.5% of SSD writes receive no subsequent SSD read, versus 44.3% in the higher-load setting.Source: §3.1 · p. 4

3.2 DRAM Competition under Full Promotion

  • Under higher load, half promotion lowers P10 transfer time from 252 to 231 ms and P5 from 310 to 277 ms versus full promotion, while lowering mean time from 81 to 78 ms.In the 2-hour replay, DRAM block evictions rise from 59,589 with zero promotion to 97,097 with half promotion and 175,314 with full promotion.Source: Figure 4 · p. 4, §3.2 · p. 5

4 SARIA

SARIA jointly manages DRAM and SSD using reuse and eviction history to selectively persist KV cache and coordinate mixed-source reloads. Its architecture combines context metadata, selective post-prefill promotion, and paced transfers to address limited SSD endurance and reload bandwidth.

4.1 System Overview

  • SARIA places new KV blocks in DRAM, uses history to select SSD persistence, and coordinates tier-spanning reloads and post-prefill promotion.Unified metadata tracks block locations and builds the reload plan used by the transfer coordinator.Source: §4.1 · p. 5

4.2 Persistence-Aware SSD Admission

  • Repeated DRAM evictions and a reuse-count threshold gate SSD persistence, trading lower write volume against delayed protection and possible recomputation on early returns.Eligible blocks can be written asynchronously while resident in DRAM, moving some SSD traffic off the reclamation path.Source: §4.2 · p. 6

4.3 Unified Metadata and Reload Planning

  • Radix-tree context metadata shares admission history across hybrid KV objects, preserves it across evictions, and builds ordered prefix reload plans across DRAM and SSD hits.This addresses differing histories among components of one context and prefixes whose cache objects are interleaved across tiers.Source: §4.3 · pp. 6–7

4.4 Lazy Partial DRAM Promotion

  • After prefill, SARIA promotes SSD-hit blocks into DRAM within both a recency-filtered eviction budget and a bandwidth-derived DRAM share.The target share balances DRAM and SSD reload paths; promotion is reduced when few DRAM eviction candidates meet the recency threshold.Source: §4.4 · p. 7

4.5 Coordinated Multi-Source Reload

  • Because nominal overlap can leave a disk-only completion tail, the coordinator paces DRAM while GDS runs and restores full DRAM bandwidth when no SSD work remains.Residual ingress can advance other DRAM-only requests while a request continues its SSD transfer.Source: §4.5 · pp. 7–8

5 Evaluation

The evaluation uses TraceLab and WEKA workloads, full-attention and hybrid models, and two testbeds. It examines end-to-end write and latency trade-offs, then isolates admission and reload mechanisms.

5.2 Main Results

  • On WEKA with Qwen3.5-9B, E3 stays within the one-day write budget, cuts writes 78.2% versus Full Duplicate, and lowers mean TTFT about 94.9% versus DRAM-only.E3 incurs a 78.4% increase in mean TTFT; E2 improves latency further but exceeds the one-day write limit.Source: §5.2 · p. 9
  • On TraceLab, E5 meets the 12-hour write limit for Qwen3.5-9B while retaining mean and P90 TTFT reductions of 66.5% and 63.7% versus DRAM-only.For Qwen3-8B, E3 meets the one-day budget with 80.3% fewer writes than Full Duplicate and reduces mean and P90 TTFT by 27.8% and 45.3% versus DRAM-only.Source: §5.2 · p. 9
  • Full Duplicate issues 27.2% fewer SSD read bytes than LMCache+ for both models on TraceLab, and 18.1% fewer for Qwen3.5-9B on WEKA.The comparison shows the read-traffic effect of DRAM-assisted reload.Source: §5.2 · p. 9
  • Expanding the session-start window from 10 to 40 minutes cuts SSD writes by 28.3% for Qwen3.5-9B and 23.5% for Qwen3-8B, while improving P25 bad CHAR.Eviction-based admission uses a fixed threshold; its mean TTFT varies over a narrower range than reuse-gated admission for both models.Source: §5.2 · p. 10
  • The complete reload configuration cuts mean TTFT versus Base by 6.4% for Qwen3.5-9B and 5.8% for Qwen3-8B, and P25 bad transfer time versus +Meta by 31.9% and 26.7%.The cumulative ablation adds unified metadata, mixed-source lookup, partial DRAM promotion, tier overlap, and coordination.Source: §5.2 · p. 10

5.3 Micro Experiments

  • Reuse-only R = 4 requires about 1.9 times as many writes as E2 for similar P25 bad CHAR on both models; adding R = 2 to E2 improves popular-context coverage at greater write cost.The admission signals expose write–coverage trade-offs, with lower P25 bad TTFT as more reusable KV cache is retained.Source: §5.3 · pp. 10–11
  • Under E2 admission, unified metadata and mixed-source lookup reduce mean TTFT by 25.4% for Qwen3.5-5.6B and 23.5% for Qwen3-1.7B, while improving P25 bad CHAR.The comparison holds promotion, overlap, coordination, and cache capacities fixed; unified planning recovers interleaved DRAM–SSD–DRAM prefix hits.Source: §5.3 · p. 11
  • Promotion cuts SSD reads, transfer time, and TTFT; coordination reduces completion time on single-SSD configurations but increases it with two SSDs.For Qwen3.5-9B at r = 0.6, reductions are 43.8%, 17.9%, and 3.5%, respectively; for Qwen3-8B at r = 0.8, they are 61.7%, 31.6%, and 9.0%. Coordination reduces completion time by 18.0% and 20.7% on slower and faster single SSDs, but increases it by 15.8% with two SSDs.Source: §5.3 · pp. 11–12

6 Related Work

Related work spans KV-cache organization, SSD-backed caching and flash admission. SARIA is distinguished by targeting SSD endurance and limited parallelism through session-reuse-aware persistence and mixed-source reload.

KV cache management

  • SGLang, HiCache, Unified Radix Cache, Marconi, and SAECache address prefix organization, hybrid-model state, or admission, while SARIA explicitly targets SSD endurance using reuse and eviction history across DRAM residencies.The passage preserves the works’ bibliography markers: SGLang, HiCache, Unified Radix Cache, Marconi [20], and SAECache.Source: §6 · p. 12

SSD-backed KV caching

  • SARIA targets deployments with limited SSD endurance and parallelism, using partial DRAM promotion and coordinated mixed-source reloads to reduce dependence on the bandwidth-limited SSD path without excessive DRAM competition.Source: §6 · p. 12

SSD Admission and Write Endurance

  • Flashield, CacheSack, and Baleen optimize general storage admission, whereas SARIA uses agent-session reuse across DRAM residencies, eviction history, and a reuse threshold to guide SSD persistence.The comparison frames SARIA’s admission signal around reuse that extends beyond a DRAM residency.Source: §6 · p. 12

7 Conclusions

SARIA addresses SSD write endurance and reload-bandwidth constraints with selective persistence, partial DRAM promotion, and coordinated transfers. Across the stated workloads and model types, it reduces writes versus full duplicate caching and mean TTFT versus DRAM-only caching.

7 Conclusions

  • Across TraceLab-derived workloads and full-attention and hybrid-attention models, SARIA reduces SSD writes by 57.1–58.9% versus full duplicate caching and mean TTFT by 42.3–79.7% versus DRAM-only caching.The conclusion attributes these outcomes to Persistence-Aware SSD Admission, Lazy Partial DRAM Promotion, and coordinated multi-source reload.Source: §7 · p. 12
Loading 2610.31169v1…