Source-linked AI summary
Lachesis: Lifetime-Aware KV Cache Placement for Agent Serving across HBM and High-Bandwidth Flash
Jaehoon Yang, Jeongmin Lee, Haneul Park, Seung Yul Lee, Nam Sung Kim, Jae W. Lee
TL;DR
Agent sessions retain KV across requests and sub-agents, increasing memory pressure, while HBF’s write endurance limits its use for dynamically generated KV. Lachesis places context segments across HBM and HBF according to their harness-determined lifetimes. In trace-driven simulation, it extends HBF lifetime over HBM-first placement, and HBF outlasts a five-year warranty on the multi-agent trace at a tight SLO.
Problem
Agent sessions contain context segments with different lifetimes, so selecting KV for HBF by read-to-write ratio does not directly apply as it does for stateless requests.
Source: §2.3 · p. 4Method
Lachesis places harness-classified short-lived segments in HBM and ranks worker segments at spawn to allocate them across HBM and HBF.
Source: §1 · p. 3, §3.5 · p. 7Results
1.19–3.13× longer HBF lifetime than HBM-first placement, with HBF outlasting a five-year warranty on the multi-agent trace at a tight SLO.
Source: §7 · p. 12Takeaways & Limitations
Placing shorter-lived KV in HBM lets it absorb more session writes and sends less data to HBF, extending flash endurance.
Source: §1 · p. 1Worker lifetime is not fixed in advance; its rank is predicted from metadata recorded when the harness spawns the worker.
Source: §3.1 · p. 5
Abstract
from arXiv · showhide
Large language model (LLM) serving is increasingly dominated by agentic workloads, in which agents and their sub-agents accumulate context as KV cache across many requests, consuming substantial memory. High-bandwidth flash (HBF) is a promising solution, providing an order of magnitude greater capacity at HBM-class read bandwidth, but its finite write endurance is the key limiting factor. Our key insight is that KV cache should be placed across HBM and HBF by its lifetime. Placing shorter-lived data in HBM lets HBM absorb more of an agent run's writes and sends less of them to HBF. As the lifetime of KV cache in agentic serving is dictated by the harness, the program that orchestrates the agents, we analyze its behavior and identify three axes along which lifetime diverges, temporal, structural, and inter-worker. Guided by these observations, we present Lachesis, a lifetime-aware KV cache placement layer between the agent harness and the serving engine. At write time, it places each segment in HBM or HBF according to its lifetime, and frees its blocks once the segment is no longer read. In trace-driven simulation, Lachesis extends HBF lifetime by 1.19-3.13x over HBM-first placement, reaching 3.3-12.2 device-years. Even under continuous 24x7 operation at the full load a tight SLO admits, HBF outlasts its five-year warranty on the multi-agent trace.
1 Introduction
Agent sessions keep KV cache live beyond individual requests, creating memory pressure, while HBF offers greater capacity but is constrained by write endurance. Lachesis places KV according to harness-determined lifetimes, extending HBF lifetime over HBM-first placement in simulation.
1 Introduction
- Agent sessions retain context beyond the request that wrote it, and concurrent sub-agents push live KV volume far beyond conventional serving.This memory pressure can force deployments to add accelerators or serve fewer sessions, raising hardware cost per request.Source: §1 · p. 1
- Placing shorter-lived KV blocks in HBM lets it absorb more session writes and sends less data to endurance-limited HBF.The placement requires knowing a block’s lifetime before it is written.Source: §1 · p. 1
- Reasoning segments hold 66–83% of a turn’s KV but live one turn, while output segments live 7.6–17.2× longer.The harness strips reasoning at the next message and carries the output forward.Source: §1 · p. 2
- Sub-agent segments produce most turn-level KV while lead-agent segments live 4.0–6.1× longer; worker lifetimes spread 3.7–23.6× between p10 and p90.Worker completion times are not fixed in advance, but their relative lifetime order can be predicted at spawn.Source: §1 · p. 2
- 1.19–3.13× longer HBF lifetime than HBM-first, with HBF outlasting its five-year warranty on the multi-agent trace at a tight SLO under continuous 24×7 operation.The comparison covers trace-driven simulation.Source: §1 · pp. 2–3
- Lachesis uses a deterministic path for harness-known temporal and structural lifetimes and a predictive path that ranks worker segments at spawn.The harness reports segment boundaries, where Lachesis frees the segment’s blocks.Source: §1 · p. 2
2 Background and Motivation
Agentic sessions accumulate context across turns and concurrent sub-agents, intensifying HBM capacity pressure. HBF offers greater capacity, but its finite write endurance motivates placing KV cache according to segment lifetime.
2.1 Agent Serving Intensifies the Capacity Wall
- Agentic sessions accumulate context across rounds and can hold a lead-agent context plus one context per running sub-agent, increasing HBM demand.Peak agentic context lengths are 29× those of conventional chatbot workloads; existing reductions and eviction strategies do not eliminate the capacity wall without trade-offs.Source: §2.1 · p. 3
2.2 High-Bandwidth Flash: Capacity Opportunity, Endurance Constraint
- HBF can relieve the capacity wall, but short-lived generated KV can quickly exhaust its finite write endurance.Generated KV grows with context length and resident sessions, making endurance a first-order challenge for agentic serving.Source: §2.2 · pp. 3–4
2.3 Why Agent Sessions Require Lifetime-Aware KV Cache Placement
- Because stateful sessions contain segments with different lifetimes, HBF placement should be decided per segment, keeping short-lived segments in HBM to reduce flash writes.The high read-to-write placement criterion for stateless requests does not directly apply when harnesses discard some context while retaining other context.Source: §2.3 · pp. 4–5
3 Context-Lifetime Analysis for Agent Serving
Agent harnesses create context segments with distinct lifetimes across temporal, structural, and inter-worker axes. These differences are measurable from harness boundaries and spawn metadata, and matter because short-lived segments can be separated from long-lived ones before placement.
3.1 Analysis Overview
- The harness fixes segment class before writing: temporal and structural splits are deterministic at turn end and sub-agent spawn, while worker lifetime is predictable from spawn metadata.Reactive LRU cannot assign the class before subsequent accesses without later migration.Source: §3.1 · p. 5
3.2 Methodology
- The methodology isolates turn-end and sub-agent-return boundaries, then measures normalized context lifetime on a fixed reference clock rather than server wall time.Tool-call latency and user delay are modeled because they depend on the deployment environment.Source: §3.2 · pp. 5–6
3.3 Temporal Axis: Reasoning-Content Stripping
- 66–83% of a turn’s written tokens sit in the reasoning-laden segment, while reasoning-stripped context outlives it by a median 7.6–17.2×.The lifetime distributions have 0.96–0.99 separability.Source: §3.3 · p. 6
3.4 Structural Axis: Lead-Agent/Sub-Agent Isolation
- Lead-agent segments outlive worker segments by a median 4.0–6.1× and reviewer segments by 4.7–38.3×, with separability 0.90–1.00.The lead-agent’s waiting period accounts for 71–88% of its lifetime, so reuse should not proxy lifetime for HBF placement.Source: §3.4 · p. 7
3.5 Inter-Worker Axis: Lifetime Heterogeneity and Spawn-Time Predictability
- Worker T_norm varies 3.7–23.6× from p10 to p90, write volume varies 2.8–22.5×, and the two correlate at Kendall’s τ of 0.51–0.72.Sub-agent segments share a class but vary by return time, so treating workers as homogeneous misses this spread.Source: §3.5 · p. 7, Figure 6 · p. 7
- Kendall’s τ is 0.53; selecting one worker in ten captures 62.8% of the shortest-lived tenth and none of the longest-lived tenth.A ridge-regression model ranks workers from spawn-record instruction, context, tools, and session position, evaluated with session-grouped five-fold cross-validation.Source: Figure 7 · p. 7, §3.5 · pp. 7–8
4 Lachesis Design
Lachesis uses harness metadata to choose HBM or HBF for each segment before writes, then frees that segment’s blocks when its boundary closes.
4 Lachesis Design
- Lachesis sits between the harness and serving engine, using deterministic segment classes, spawn-time worker ranks, and a live-segment table to choose tiers before writes.The design overview shows the layer receiving segment IDs, spawn records, and context boundaries from the harness.Source: §4 · p. 8, Figure 8 · p. 8
- The deterministic path places short-lived segments in HBM and long-lived segments in HBF; reviewers take HBM unconditionally, while workers use predictive placement.Turn-end boundaries separate reasoning-laden from reasoning-stripped context; sub-agent spawn separates sub-agent from lead-agent context.Source: §4 · p. 8
- The predictive path ranks workers by spawn-time lifetime prediction, admits the first k to HBM, and allocates the rest in HBF.Reviewer capacity is reserved first, and k depends on HBM free capacity rather than a fixed parameter.Source: §4 · p. 8
- A segment-table row tracks each live segment’s class, closing boundary, and block IDs; Lachesis frees those blocks at the boundary using tier-specific free lists.The engine interface consists of allocation from a named tier and freeing block IDs back to their original tier’s list.Source: §4 · p. 9
5 Evaluation
The evaluation uses simulated serving-engine replays of agent traces to compare Lachesis with HBM-first placement across memory configurations, models, and SLOs. Lachesis extends HBF lifetime in multi-agent and single-agent workloads, while ablations and write-volume results examine how its placement paths contribute.
5.1 Implementation; 5.2 Experimental Setup
- The replay evaluates two models across five GPU-memory configurations and two SLOs for the multi-agent trace, and uses tight SLO alone for single-agent traces.The simulator replays real agent traces through a simulated serving engine with two-tier memory; HBM-first places blocks in HBM when free and otherwise in HBF.Source: §5.1 · p. 9, §5.2 · p. 9
5.3 Multi-Agent Results
- 3.3–9.3 device-years versus HBM-first’s 1.9–4.8 at four split-tier configurations, with HBF lifetime extended 1.50–2.95× at tight and 1.19–1.89× at loose SLOs.At tight SLO, HBF outlasts the five-year warranty under Lachesis at every configuration and both models; HBM-first does so only with Qwen3-235B-A22B on H3’s daisy chain.Source: §5.3 · p. 9, Figure 9 · p. 10
- Lachesis sends 19–51% of multi-agent run write bytes to HBM, versus 1–7% under HBM-first placement.HBM-first frees blocks only under memory pressure, while Lachesis places the lead-agent segment in HBF and admits predicted short-lived workers to HBM.Source: §5.3 · p. 10, Figure 10 · p. 10
5.4 Ablation Study
- Without worker ranking, Lachesis reaches 1.11–2.00× HBM-first lifetime; adding the predictive path contributes a further 1.33–1.54× on split-tier configurations.The predictive path’s gain is 1.33–1.54× with a reviewer present versus 1.05–1.14× without one.Source: §5.4 · p. 11
5.5 Single-Agent Results
- 4.6–12.2 device-years versus HBM-first’s 2.1–6.4 on single-agent traces, extending lifetime by 1.35–3.13× across four split-tier configurations.These traces exercise the temporal axis alone; Lachesis’s lifetime peaks at one to three HBM slots in eight, then falls as added slots remove HBF stacks without saving further flash writes.Source: §5.5 · p. 11
6 Related Work
Prior HBF designs either avoid writing generated KV or address the cost and placement of writes using criteria other than lifetime. Lachesis instead uses the harness-determined lifetime of each context segment to decide placement.
6 Related Work
- Early HBF designs restrict flash to read-only data; H3 stores model weights and precomputed KV in HBF while keeping generated KV in HBM, and FLINT supports read-only placement.Neither design spends flash endurance on generated KV because neither writes it.Source: §6 · p. 11
- FlashAccel, Park et al., and DASH write generated KV to flash but address write cost or placement using mechanisms other than data lifetime [47],, [20].DASH bases placement on size, mutability, and reuse.Source: §6 · p. 11
- Lachesis decides from the lifetime the agent harness fixes before a context segment is written, rather than from read-to-write ratio or output length.The related-work discussion distinguishes this segment-lifetime criterion from prior criteria.Source: §6 · pp. 11–12
7 Conclusion
Lachesis places agent-serving KV across HBM and HBF according to harness-defined context-segment lifetime. In trace-driven simulation it extends HBF lifetime over HBM-first placement, and HBF outlasts its five-year warranty on the multi-agent trace at tight SLO, even under continuous 24×7 operation.
7 Conclusion
- Agent-session context divides at harness-defined boundaries into segments whose lifetimes differ along temporal, structural, and inter-worker axes.Each segment’s lifetime is knowable before it is written.Source: §7 · p. 12
- Lachesis places each segment according to its lifetime using a deterministic path for harness-fixed segment classes and a predictive path that ranks worker segments at spawn.Source: §7 · p. 12
- Lachesis extends HBF lifetime by 1.19–3.13× over HBM-first placement, and HBF outlasts a five-year warranty on the multi-agent trace at tight SLO under 24×7 operation.Source: §7 · p. 12
A Workload Corpus Statistics
The workload corpus combines temporal-regime single-agent traces and structural-regime multi-agent benchmarks, with harness and prompt choices varying by regime. The corpus includes failed and timed-out sessions.
A Workload Corpus Statistics
- Temporal-regime benchmarks use Harbor’s official prompts, while structural-regime benchmarks use official prompts and reference implementations ported to AOrchestra.Source: A Workload Corpus Statistics · p. 14
- Tokens processed counts each request’s input and output, including reused context at each round; tokens written counts only newly materialized content.Structural traces report per-segment sizes rather than per-request lengths, so their table reports only tokens written.Source: A Workload Corpus Statistics · p. 14
- Neither corpus excludes sessions that failed or timed out.Source: A Workload Corpus Statistics · p. 14
B Trace Fields, Clock Charges, and KV Accounting
This appendix specifies the trace fields and clock charges used to model agent requests, then details segment-level KV read/write accounting. It clarifies which prefill and decode activity contributes to those quantities.
B Trace Fields, Clock Charges, and KV Accounting
- The trace records request lineage, tokenized inputs and outputs, tool-call and observation boundaries, and cache fields; multi-request segments become live at their first write.Requests are aligned in session order to distinguish reused KV in the matched prefix from newly materialized KV in the unmatched suffix and generated tokens.Source: B Trace Fields, Clock Charges, and KV… · p. 14
- TTFTref is charged once per observation, while tool-call latency is modeled exponentially using the published mean for the calling tool’s class.A prefill deployment provisioned to the reference point completes any single observation’s prefill within TTFTref.Source: B Trace Fields, Clock Charges, and KV… · p. 14
- Table 3 reports corpus scale per workload.Source: Table 3 · p. 15
- For each request, the accounting tracks the segment’s KV tokens present before each decode step and considers requests whose decode stage reads that segment.The full-context-attention setup defines the per-step token count; Q(s) is the set of requests whose decode stage reads s.Source: B Trace Fields, Clock Charges, and KV… · p. 15
- Segment read volume counts only decode-stage reads, while write volume counts initial prefill tokens; initial prefill time is excluded from Tnorm.Observation-induced partial prefill is accounted for in Tnorm through TTFTref, not in segment read volume.Source: B Trace Fields, Clock Charges, and KV… · p. 15