Understanding KV Residency in Production-Scale Agentic LLM Workloads

1 Introduction

• Agentic loops repeatedly invoke models and tools, append results, and continue until the task is complete, unlike a short request-level serving sequence.A session comprises one or more turns sharing a session identifier and accumulating context.Source: §1 · p. 2
• Prior agent-trace studies do not expose the serving deployment selected for each call, limiting measurement of how deployment changes affect KV reuse.At hyperscaler scale, errors in reuse, residency-lifetime, or locality estimates can translate into terabytes of excess GPU memory or extensive context reconstruction.Source: §1 · p. 2
• Agentic workloads have median session depths of 13 calls and P99 depths of 160 in Research, versus 3 and 52 in Chat, with aggregate KV reuse of 0.50–0.80 versus 0.27.All four workloads have median within-session idle gaps of 6–10 seconds; Code’s interval reflects tool-execution cadence.Source: §1 · p. 3
• The paper examines reclaiming state after one idle hour and replaying residency under constrained memory to separate capacity-driven recomputation from routing-driven recomputation.The analysis quantifies potential value from affinity-aware routing and multi-resolution KV retention.Source: §1 · p. 3
• Because production traces cannot be released, the authors plan a parametric trace generator to enable further study.The planned release is AgenticTraceSim, described with a residency replay framework in the paper context.Source: §1 · pp. 3–4
• The four Microsoft M365 traces span three agentic applications and Enterprise Chat, with more than 121 million model calls and deployment structure for three workloads.The traces combine session, token, and model structure with serving region and deployment for three workloads.Source: §1 · p. 3
• A P99 Research session requires 81.6 GiB under the reference KV representation, exceeding an H100’s 80 GB HBM; holding all live sessions at peak requires an estimated 501.8 TB.Research reuse is also associated with deployment continuity: 88.9% when consecutive calls stay on one deployment versus 29.0% after a same-model deployment change within the region.Source: §1 · p. 4
• At 10% of the busiest-hour current-context peak, FNU leaves 31.1% of Research calls to recompute context, while int4 MRR reduces that rate to 13.6%.With unlimited memory, modeling KV as deployment-local raises the first-touch floor from 5.4% to 20.2%.Source: §1 · p. 4

2.1 What Prior Characterizations Miss

• Prior characterizations do not jointly examine diverse production enterprise workloads, per-call deployment, byte-valued KV residency, and fleet-scale capacity-constrained replay.The paper’s replay separates memory-driven recomputation from deployment-local placement effects and evaluates lower-precision retention.Source: §2.1 · p. 5

2.2 Four Production M365 Workloads

• The four M365 traces span three agentic applications and Enterprise Chat, with telemetry covering context, cached tokens, outputs, timestamps, and model identity.Code records session events rather than model calls and is excluded from byte-valued working-set and placement analyses.Source: §2.2 · pp. 5–6

2.3 Measurement Methodology

• KV-byte estimates use Llama-2-70B geometry as a public stand-in, so absolute byte values scale linearly with the assumed bytes per token.Reuse and context-share measures are independent of that assumption.Source: §2.3 · p. 7

2.4 How Agentic Is Each Workload?

• Over 89% of Research calls are application-issued, while Chat also has a majority of calls without a new user turn but much shorter loops.Assist’s depth suggests a mixture of single-call and longer sessions, but its internal orchestration is not measured directly.Source: §2.4 · pp. 7–8

2.5 Use of generative AI

• Generative AI tools helped refine language and presentation and implement analysis code, but did not generate the study’s ideas, telemetry, or conclusions.The authors reviewed assisted edits and code and accept responsibility for the manuscript and artifacts.Source: §2.5 · p. 8

2.6 Privacy Considerations

• The analysis uses aggregate operational metadata, excludes message content and production identifiers, and does not release confidential traces.The public artifact uses aggregate fitted parameters and synthetic traces.Source: §2.6 · p. 8

• Repeated model re-entry within turns accumulates context across calls, creating sustained demand for reusable KV state.Source: §3 · p. 8

• Research sessions have median depth 13 and P99 depth 160 calls, versus 3 and 52 for Chat; Assist has median depth 1 and P90 depth 28.Source: §3.1 · p. 8, Figure 3 · p. 9
• Code is reported separately in events, mostly tool executions: its median session has 14 events and its P99 session has 963.Code is not plotted because its trace records events rather than model calls.Source: §3.1 · p. 8

• Research’s median context grows from 9,764 tokens on call 1 to 41,045 by call 11 and 68,118 by call 30.It first reaches 95% of its call-30 value at call 27.Source: §3.2 · p. 9
• 90.9% of multi-call Research sessions have at least one context reduction, and the median reduction leaves 23.7% of the preceding call’s context.Telemetry does not record the cause; reductions may reflect compaction, summarization, truncation, or model-role transitions.Source: §3.2 · p. 9

3.3 Context, Not Generation, Dominates Token Volume

• At least 98% of combined input and output token volume is context across all four traces.Context share measures KV state carried by calls, not prefill compute spent on uncached context.Source: §3.3 · p. 9

3.4.1 Call-context KV concentrates on the Spine.

• The Spine carries 67% of Research call-context KV under the conservative mapping and 90% under the broader mapping.The class boundary is definition-dependent, but the heavyweight class dominates under both mappings.Source: §3.4.1 · p. 10, §3.4 · p. 10

3.4.2 Call-context bytes and uncached prefill have different class distributions.

• The Spine holds 67% of call-context KV bytes but only 32% of uncached-context tokens, so most Spine context is served from the prefix cache.Losing that cached prefix would turn a large reused context into costly recomputation.Source: §3.4.2 · p. 10

3.4.3 Model-class transitions are predictable.

• ≈88% of observed Research model-class transitions are predicted by choosing the current class’s most frequent successor.Spine self-transitions with probability 0.97; this signal is available in multi-class Research, not single-model Assist.Source: §3.4.3 · p. 10

3.4.4 Reuse declines after model-class excursions.

• 92.5% mean reuse for consecutive Spine calls falls to 45.1% after one intervening non-Spine call and 30.1% after three or more.The analysis does not control for elapsed time or deployment placement, so this is an association rather than evidence of causal interference.Source: §3.4.4 · pp. 10–11

3.5 Reuse Is Bimodal at the Call Level

• 51.9% of Research calls achieve at least 95% reuse, while 47.6% of Chat calls have reuse below 20%.Only 40.7% of Research sessions have aggregate reuse of at least 0.8, since sessions mix cold calls with warm continuations.Source: §3.5 · p. 11
• 29.9% of Assist calls have intermediate reuse of 20–80%, the largest such share among the workloads described.Per-call reuse is bimodal overall, but Assist is more evenly distributed across reuse levels.Source: §3.5 · p. 11
• A residency policy must combine reuse with session continuation, working-set size, and fleet memory pressure.Realized reuse also depends on the serving system retaining and recovering context.Source: §3.5 · p. 12
• Research has aggregate reuse of 0.80 and a context-carry tax of 5.1, meaning it processes 5.1 context tokens per uncached context token prefetched.The tax is close to TraceLab’s 5.3× prefill amplification, although the denominators differ.Source: §3.5 · p. 12

4.1 A Single Session Can Fill a GPU

• A Research session’s estimated KV working set reaches 81.6 GiB at P99, exceeding an H100’s 80 GB HBM.KV state alone is estimated; model weights, activations, and runtime state require additional memory, and a tensor-parallel group may collectively hold a session’s KV.Source: §4.1 · p. 12, Figure 7 · p. 13

4.2 Across the Fleet, Live Sessions Add Up

• 501.8 TB is the exact estimated peak fleet KV residency ceiling for live Research sessions, with 11,358 concurrent sessions at peak.Hourly bins instead yield 709.4 TB, a 1.41× overstatement.Source: §4.2 · p. 13

4.3 Retained State Sits Idle Across Heavy-tailed Gaps

• Research idle gaps have an 8.38-second median but a P99 of approximately 20 minutes, and the longest observed gap is 7.9 days.In Research, inter-turn gaps are 7.0% of gaps but account for 92.6% of total idle time.Source: §4.3 · p. 13, Figure 8 · p. 13

4.4.1 Long Research sessions span multiple deployments.

• Only 45.0% of Research sessions are single-deployment, and 57.6% are single-region, compared with 80.2% and 83.3% for Assist.A Research session touches a mean of 4.48 deployments across 3.04 regions.Source: §4.4.1 · p. 14
• The Research single-region and single-deployment subsets are structurally different from the full workload, with median depth falling from 13 calls overall to 5 and 3.Their aggregate reuse and capacity values therefore cannot be interpreted as causal effects of a placement policy.Source: §4.4.1 · p. 14

4.4.2 Reuse is strongly associated with deployment continuity.

• 88.9% mean reuse on same-deployment Research continuations falls to 29.0% after a same-model move within a region and 23.5% after a cross-region move.The analogous Assist values are 64.8%, 27.7%, and 7.5%, respectively; the consistent decline is associated with deployment continuity.Source: §4.4.2 · p. 14
• Chat’s rare same-model deployment and region moves are selection-confounded because those categories can differ in session depth and composition.The deployment-continuity finding is based on Research and the single-model Assist control.Source: §4.4.2 · p. 14

4 Why Not Keep Every Session’s KV Resident?

• A one-hour idle cap reduces the exact Research fleet ceiling from 501.8 to 140.8 TB while raising reconstruction from 4.07% to 4.53%.The penalty rises more rapidly when the cap enters the short-gap body of the distribution.Source: Figure 10 · p. 15

5.1 Reclaiming Idle State: Nominally Live State Is Not Always Residency-worthy

• A one-hour idle cap cuts the Research hold-all-live ceiling 3.6×, from 501.8 to 140.8 TB, while reconstruction rises from 4.07% to 4.53%.At a 15-minute cap, the ceiling falls to 97.2 TB and reconstruction reaches 5.2%.Source: §5.1 · p. 16

5.2 Capacity Limits Destructive Eviction

• At a 4.7 TB pooled budget, FNU reconstructs 31.1% of Research calls, versus 62.9% for LRU; FNU is a heuristic for variable-size KV objects, not an optimum.The busiest-hour replay covers 176,007 calls from 9,486 sessions and has a 5.4% first-touch floor.Source: §5.2 · p. 16

5.3 Retaining a Lower-precision Copy: Multi-resolution Retention

• At 10% of the busiest-hour peak, int4 MRR reconstructs 13.6% of Research calls, versus 31.1% for FNU and 62.9% for LRU.MRR retains a recoverable int4 copy in the same pooled budget; its advantage over LRU reflects retaining smaller copies, not future knowledge.Source: §5.2 · p. 16, §5.3 · p. 17, Figure 11 · p. 17
• At the tightest evaluated budget, 0.02 K⋆hr, FNU reconstructs 67.60% of calls versus 72.92% for MRR; host-tier demotion lowers reconstruction further.A host tier equal to the pooled HBM budget gives 9.46%, while a tier twice as large gives 5.57%; omitted transfer and promotion costs qualify these replay results.Source: §5.3 · pp. 17–18

5.4 Policy Separation Differs Across Workloads

• At a 10% pooled budget, Assist’s LRU, FNU, and MRR reconstruction rates are 12.6%, 11.9%, and 11.1%, while Chat’s are 40.7%, 28.4%, and 22.5%.Chat’s 9,749 TB hold-all-live ceiling over 24 hours comes from concurrency, whereas deep agentic sessions create large individual working sets.Source: §5.4 · p. 18

5.5 Routing: Deployment-local KV Creates an Independent Floor

• With unlimited capacity, deployment-local KV raises Research’s first-touch floor from 5.4% to 20.2%, because sessions routed among deployments can create separate cache objects.The replay assumes KV is not automatically shared across deployments; lowering this floor requires affinity, replication, or migration.Source: §5.5 · p. 18

5.6 Lessons learned

• Across the section’s lessons, a one-hour idle cap cuts the ceiling from 501.8 to 140.8 TB, FNU reconstructs 31.1% of calls, and int4 MRR reconstructs 13.6%.Mean reuse falls from 88.9% to 29.0% after a same-model deployment change.Source: §5.6 · p. 19

5.7 Open Problems and Limitations

• The authors frame reversible KV retention and application-aware placement as research directions, not evaluated system designs.The four Microsoft M365 workloads were observed during specific windows, and results depend on choices including liveness definition, cache-object key, and pooled replay budgets.Source: §5.7 · pp. 19–20

6 AgenticTraceSim: Reproducible Agentic Traces

• Production traces contain sensitive serving telemetry and cannot be released without risking client and operational confidentiality.The planned release is a parametric generator and replay framework for synthetic or user-supplied streams.Source: §6 · pp. 20–21
• AgenticTraceSim models arrivals, session depth, context growth, model-class transitions, per-call reuse, and idle gaps, coupling components to preserve residency-relevant dependencies.The planned release includes SR and SD population profiles, which do not model deployment transitions or placement policies.Source: §6 · p. 20
• For depth, idle gaps, and per-call reuse, generated distributions have Kolmogorov–Smirnov distance at most 0.10 from measured Research distributions.Generated P50/P90/P99 working sets are 21.6/55.9/82.5 GiB versus 25.7/59.2/81.6 GiB measured; depth–peak-context and depth–duration rank correlations are reproduced within 0.05.Source: §6 · p. 20
• Matched measured and synthetic workloads retain the reconstruction-rate ordering LRU ≥ FNU ≥ unbounded-tier MRR at every evaluated budget.The measured and generated LRU-to-MRR improvements are 2.45× and 2.50×, respectively, with overlapping bootstrap intervals.Source: §6 · pp. 20–21, Figure 12 · p. 21
• A marginals-only generator fails to reproduce the measured LRU-to-MRR improvement, indicating that modeled workload dependencies matter for policy evaluation.The generator reproduces shapes, ratios, and orderings, not absolute fleet magnitudes; deployment-local placement results require production traces.Source: §6 · p. 21

Agentic workload characterization

• Prior agentic workload studies characterize repeated calls, growing contexts, and reuse, while this work focuses on multi-call production sessions and deployment placement.The discussion cites TraceLab, the GitHub Copilot study, and the ReAct study [66] [28] [60].Source: §7 · p. 21

LLM serving and KV management

• Serving systems provide KV management, prefill/decode scheduling, prefix caching, and distributed KV transport; this work measures the session- and fleet-level demand those mechanisms serve.Examples include PagedAttention and vLLM [20], Sarathi-Serve, and Splitwise, DistServe, and Mooncake [40] [65].Source: §7 · p. 22

Agent-aware serving and orchestration

• Agent-aware systems use application semantics for prefetching, cache management, scheduling, and allocation, while this work quantifies workload structure and separate reconstruction costs from memory and routing.The discussion also identifies per-request keep, discard, or swap decisions during tool waits [1] [22].Source: §7 · p. 22

Context reduction and external memory

• Context reduction changes what enters the model context, whereas this work characterizes how the resulting KV state is reused, retained, and placed across a serving fleet.The paper discusses how context reduction and KV residency can compose in Section 5.7.Source: §7 · p. 22

KV compression and tiering

• Prior work evicts, quantizes, or tiers KV; this work’s trace-driven replay asks how much reconstruction remains under destructive eviction and how much lower-precision retention removes.The discussion treats these techniques as complementary: prior work supplies representations and tiers for displaced KV.Source: §7 · p. 22

8 Conclusion

• Deep agentic sessions create large individual working sets, while enterprise chat creates substantial fleet demand through concurrency.The conclusion reports this distinction within its characterization of four Microsoft M365 workloads spanning three agentic applications and enterprise chat.Source: §8 · p. 22
• A one-hour idle cap lowers the Research hold-all-live ceiling by 3.6× while raising replayed reconstruction from 4.07% to 4.53%.The conclusion places this idle reclamation result alongside the separate capacity and placement findings.Source: §8 · p. 22
• At 10% of the busiest-hour current-context peak, int4 MRR reconstructs 13.6% of Research calls versus 31.1% for FNU; deployment-local KV raises the unlimited-memory floor from 5.4% to 20.2%.The conclusion distinguishes limited-memory eviction from deployment changes that make state unavailable.Source: §8 · p. 23
• Because production traces contain sensitive telemetry, the authors plan to release AgenticTraceSim and a replay framework for synthetic or user-supplied call streams.The generator preserves dependencies needed to reproduce the reconstruction-rate ordering without exposing production events.Source: §8 · p. 23