Efficient Agentic LLM Serving over SSD-based Sparse KV Storage

1 Introduction

• Agentic workloads combine growing histories and repeated append prefill with costly DRAM-centric KV storage, while SSD-centric storage faces fragmented accesses and read-write interference.Concurrent SSD writes can reduce read bandwidth by nearly 60%; multi-round read volume can be tens of times write volume.Source: §1 · pp. 1–2
• Sparse attention makes KV demand known only after indexer execution, so exact SSD reads can land on the inference critical path and occupy over 30% of append prefill time.Dense attention permits straightforward prefetching because all history KV caches are consumed.Source: §1 · p. 2
• Janus reuses the model’s own indexer on earlier intermediate states to predict KV demand, prefetches predicted caches during computation, then fetches misses before attention for lossless outputs.The prediction is training-free; prediction work is partitioned across GPUs to avoid delaying inference.Source: §1 · p. 2
• Janus coalesces adjacent selected reads, packs scattered GPU KV pages into sequential CPU-SSD writes, and caps background writes during read phases.These optimizations target fragmented I/O and read-write interference.Source: §1 · p. 2
• Janus outperforms state-of-the-art baselines by up to 1.57–3.69× in time-to-first-token latency, with SSD I/O exposed on the critical path below 6.5% of append prefill latency.The evaluation uses multiple agentic workload traces and long-context sparse-attention models.Source: §1 · p. 2

2.1 LLM Inference and Serving

• Append prefill processes newly appended tokens while reusing cached history; recomputing that history each round would waste GPU cycles.Prefill is typically compute-bound, whereas decode is typically memory-bound and often uses continuous batching.Source: §2.1 · p. 3

2.2 Sparse Attention

• A Transformer layer comprises attention and a position-wise FFN, while frontier models use interleaved sparse-dense layers or IndexShare groups that reuse selected indices.DeepSeek-V4 implements selection at entry granularity, although the description uses token granularity.Source: §2.2 · pp. 3–4, Figure 1 · p. 4
• Sparse attention uses an indexer to select top-k history tokens for each appended token, unions those selections, and loads the selected history KV caches before attention.The indexer also uses appended-token hidden states and stored history indexer keys; selected indices are represented per token.Source: §2.2 · p. 3

2.3 KV Storage

• KV caches and indexer keys persist across a session and may be offloaded to CPU DRAM or SSD; NVMe queue pairs support asynchronous GPU-submitted I/O that overlaps computation.GPU-initiated storage systems expose NVMe queues to GPU kernels, while CPU setup and coarse-grained metadata management remain outside the data path.Source: §2.3 · p. 4

Length characteristics of agentic workloads

• In three agentic traces, median histories reach 77K, 131K, and 70K tokens, while append queries are 3–20× outputs in median and P99 values.The traces combine long inputs, short outputs, and many rounds.Source: §3 · p. 5, Table 2 · p. 5

Observation 1: Append prefill dominates KV loading

• Across three models, roughly 99% of history KV caches needed during decode are already loaded in append prefill.Most decode steps trigger no KV loading and reuse caches loaded during prefill or generated in that round.Source: §3 · p. 5, Figure 2 · p. 5
• Append prefill’s union of sparse selections brings a broad history working set into GPU HBM, which tends to remain resident through decode.This concentrates SSD history loading in append prefill rather than subsequent decode steps.Source: §3 · p. 5

Observation 2: Agentic sessions amplify KV reads

• Sparse-attention reads across real agentic traces remain 3.9–23.5 times write volume and grow with the number of rounds.A history KV cache written once can be selected and reloaded by multiple later append prefills.Source: Figure 3 · p. 5, §3 · p. 6

Implications for SSD-backed sparse attention

• Read efficiency matters because agentic read amplification combines scattered sparse-attention reads with fragmented writes and SSD-internal read–write contention.Concurrent SSD writes can reduce read bandwidth by nearly 60%.Source: §3 · p. 6
• The observations motivate hiding sparse KV loading during append prefill and prioritizing read efficiency when optimizing SSD I/O.Existing decode-oriented prefetchers do not directly address append prefill; prior prefill approaches described here lack both lossless execution and a sufficiently long SSD overlap window.Source: §3 · pp. 5–6

4.1 Overview

• Janus predicts sparse KV selections ahead of their target layers and overlaps prefetching with computation, fetching missed caches before attention to preserve model outputs.The lookahead reuses the target layer’s indexer on earlier hidden states without training or offline calibration.Source: §4.1 · p. 6, §4.2 · pp. 6–7

4.2 KV Selection Prediction

• With d=2 for DeepSeek-V4 and d=4 for GLM-5.2, recall exceeds 96% across all three models, while predictive-read redundancy stays below 5% for DeepSeek-V4 and 10% for GLM-5.2.At these distances, only 4% of sparse KV loading remains on the critical path; shorter distances yield diminishing returns aligned with model designs.Source: §4.2 · p. 7
• Synchronization-free prediction accelerates prediction by 2.8× on average for GLM-5.2 and 2.3× for DeepSeek-V4-Flash, with almost the same recall and a small precision drop versus non-parallel prediction.It replaces global top-k with per-GPU local top-k/N selections, enabling speculative reads without synchronization; it does not ensure exact global top-k for each appended token.Source: §4.2 · p. 8

4.3 SSD I/O Optimization

• Janus merges only selected, physically adjacent SSD blocks into read requests, reducing command count and queue pressure without fetching unselected blocks.Fully scattered selections fall back to block-granular reads.Source: §4.3 · pp. 8–9
• CPU-assisted write packing stages scattered GPU KV pages in a bounded CPU buffer, relayouts them in token order, and writes one contiguous buffer to SSD.Token-ordered placement creates contiguous regions for later best-effort read packing while avoiding GPU packing work.Source: §4.3 · p. 9
• Janus caps background writes at 512 MB/s per SSD during append prefill, preserving about 90% of read bandwidth when reads are active.The cap is enforced on 256 MB write chunks when SSD reads are in flight, prioritizing read efficiency without permanently sacrificing write throughput.Source: §4.3 · p. 9, Figure 10 · p. 9

4.4 Implementation

• Janus integrates GPU-initiated SSD I/O with vLLM, supports DeepSeek-V4 and GLM-5 sparse-attention models, and uses layer-wise prefetching plus GPU copy engines for KV gathering.The implementation comprises 44K lines of Python and C++/CUDA; SGLang is also compatible.Source: §4.4 · p. 9

5.1 Experimental Setup

• Experiments use three models and three agentic traces on an 8-H200, 8-SSD server, comparing Tutti, LMCache, and Janusbase using TTFT and TPOT.All experiments are averaged over five runs.Source: §5.1 · pp. 9–10

5.2 End-to-end Comparison

• Janus achieves up to 1.57× (1.22× on average) speedup over Tutti, 3.69× (1.85× on average) over LMCache, and 1.71× (1.27× on average) over Janusbase.It has the lowest TTFT across request clusters with different history lengths and append-to-history ratios.Source: Figure 11 · p. 10, §5.2 · p. 10
• In end-to-end serving, Janus improves P95 TTFT by 1.05–1.51× over Tutti, 1.31–3.04× over LMCache, and 1.01–1.68× over Janusbase.Competitors’ TPOT latencies differ by approximately 3%.Source: Figure 13 · p. 11, §5.2 · p. 11
• Janusbase’s post-selection KV reads consume 17.5–36.3% of append-prefill time, while Janus’s corrective reads consume 0.2–6.4%.Prediction adds 0.3–1.2% of append-prefill time across nine cases.Source: §5.2 · p. 11

5.3 Effectiveness of SSD I/O Optimization

• Disabling best-effort read packing degrades P95 TTFT by 4.2%; GPU packing and no packing degrade P95 TPOT by 3.7% and 5.5%, respectively.CPU-assisted packing provides contiguous SSD writes without competing with model execution for GPU SMs.Source: Table 3 · p. 11, §5.3 · p. 12
• Without write capping, P99 TTFT rises substantially relative to a 512 MB/s cap; a 128 MB/s cap yields P99 TTFT 16.2% and 7.3% higher than 512 MiB/s.Tight caps slow KV offloading and increase queueing; moderate caps provide sound TTFT performance without laborious tuning.Source: Figure 14 · p. 12, §5.3 · p. 12

• SSD-backed KV systems commonly assume a reusable prefix identifies required KV before prefill, whereas sparse attention reveals a query-dependent subset during layer execution.Janus coordinates sparse selection with SSD I/O rather than loading the full cached prefix.Source: §6 · p. 12
• Janus serves models with native sparse selection without altering their algorithms, preserving exact outputs while addressing limitations of predictive systems targeting decode or dense-attention models.It provides longer lookahead for append prefill and corrects prediction misses before attention.Source: §6 · p. 12
• Unlike agentic systems focused on scheduling and cache placement, Janus optimizes each append prefill’s inference path to fetch query-dependent sparse KV from SSD.Related systems exploit request dependencies, tool lifetimes, or session locality.Source: §6 · p. 12

• Janus achieves up to 1.57–3.69× speedup, or 1.22–1.85× on average, in time-to-first-token latency compared with existing works.The evaluation also characterized append prefill as the bottleneck for SSD reads in agentic workloads.Source: §7 · p. 12
• Janus speculatively prefetches KV caches using lookahead prediction and performs corrective reads to ensure lossless model outputs.It also applies SSD I/O optimizations tailored to sparse-attention agentic workloads.Source: §7 · p. 12
• The work identifies append prefill as the point where SSD read efficiency should be emphasized in agentic workloads.Source: §7 · p. 12