PaperlayerView exact revision on arXiv

Source-linked AI summary

Capture the lifecycle: KV Cache management in ReAct Agents with KVTether

Kaihua Fu, Yukun Zhou, Chaokun Chang, Yinghao Yu, Luping Wang, Guodong Yang, Jiuchen Shi, Quan Chen, Wei Wang

arXiv:2609.39819v1cs.OS

TL;DR

Long-context ReAct agents need KV reuse, but serving stacks see cache accesses rather than the message lifecycles that determine future reuse. KVTether traces harness primitives and maps their semantics to KV lifecycle states that guide cache management. Across benchmarks and production traces, it reduces estimated task pricing and request latency relative to LMCache and MORI.

  • Problem

    Agent harnesses manage message lifecycles that serving stacks cannot see, leaving KV management without the runtime semantics needed to distinguish live, waiting, and dead cache.

    Source: §1 · p. 2

  • Method

    KVTether traces message-level primitives in agent harnesses, translates their events into KV lifecycle states, reclaims dead KV, and applies MRU to live-but-idle KV.

    Source: §1 · p. 2

  • Results

    KVTether lowers average estimated task pricing by 40.0% vs LMCache and 33.2% vs MORI, and end-to-end request latency by up to 26.3% and 17.4%, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    KVTether addresses obsolete cache residency and premature eviction under uncertain reuse intervals by reclaiming dead KV and managing live-idle KV separately.

    Source: §9 · p. 13

    Exact next-use times for live KV remain uncertain because decoding, tools, subagents, queueing, and scheduling vary online; prediction cannot replace lifecycle semantics or reliably eliminate ordering errors.

    Source: §2.4 · pp. 4–5

Abstract

from arXiv · show

Efficient serving of long-context reasoning-and-acting (ReAct) agents relies on KV cache reuse to reduce large language model (LLM) prefill latency and monetary cost. However, a semantic gap exists between agent harnesses and the underlying serving stack. Through context mutation, tool execution, and subagent coordination, context messages may become actively engaged, permanently discarded, and temporarily unused, while the serving stack only observes accesses to the corresponding KV cache. This lifecycle blindness prevents recency-only policies such as LRU from reclaiming dead KV promptly and from preserving older KV that will be reused sooner than newer entries. We present KVTether, a lifecycle-aware KV cache management framework for ReAct agents. By tracing semantic primitives embedded in agent harnesses, KVTether captures runtime lifecycle semantics during highly dynamic execution. KVTether then translates message-level semantics into KV-level lifecycle states and uses these states to drive state-prioritized cache management without exposing physical complexities to agent harnesses. After reclaiming dead KV, KVTether preferentially preserves live-but-idle KV that is waiting for reuse, reducing premature eviction before reuse. Across agent benchmarks and production workloads, KVTether reduces end-to-end request latency by up to 26.3% and 17.4% relative to LMCache and MORI, respectively, and lowers estimated task cost by 40.0% and 33.2% on average.

1 Introduction

Long-context ReAct serving needs KV reuse, but cache storage sees accesses rather than message lifecycles, obscuring what should be retained or reclaimed. KVTether exports harness semantics into state-guided cache management and reports lower latency and task pricing.

1 Introduction

  • Long-context ReAct requests can reach hundreds of thousands of tokens, making KV reuse important for avoiding repeated prefill, while a 258K-token GLM-4.7 context requires 93GB of KV cache.A Codex request can reach a 258K-token context.Source: §1 · p. 1
  • Long-term messages remain reusable across calls, short-term messages until removed or replaced, and temporary messages only through the current call.These distinct logical lifetimes are not visible to the storage layer as message states.Source: Figure 1 · p. 1, §1 · p. 1
  • Harnesses manage message lifecycles through operations such as append, replace, remove, fork, and one-shot construction, but serving layers observe token sequences, cache pages, and access events.Consequently, storage cannot determine whether cached KV is live, retained, waiting, or dead.Source: §1 · p. 1
  • Across three workloads, 71.0–95.0% of LRU evictions displace KV that will be needed sooner than the newly admitted KV, while zombie KV occupies up to 36.3% of LMCache capacity.Zombie KV can persist after a compaction subagent replaces a long history with a summary.Source: §1 · p. 1
  • KVTether traces message-level primitives in diverse harnesses, translates exported events into KV lifecycle states, reclaims dead KV, and uses MRU to preserve live-but-idle KV without precise reuse prediction.Runtime tracing is needed because ReAct execution topology and lifecycle events arise dynamically in the harness; the integration interface is designed to avoid exposing physical KV-management details to application developers.Source: §1 · p. 2
  • KVTether lowers average estimated task pricing by 40.0% vs LMCache and 33.2% vs MORI, and end-to-end request latency by up to 26.3% and 17.4%, respectively.The evaluation covers agent benchmarks and production traces from a large cloud provider.Source: §1 · p. 2

2 Background and Motivation

ReAct agents build evolving message contexts across LLM calls, tools, and sub-agents, creating KV reuse that depends on runtime lifecycle and timing. Existing semantic-blind policies miss reuse and retain KV that is dead or less likely to be reused soon.

2.1 ReAct Execution and Context Lifecycles

  • ReAct contexts evolve through repeated LLM calls and tool results; compaction, one-shot subagent contexts, and inherited prefixes create distinct message lifecycles.Child contexts may evolve independently, so ending reuse in one context does not necessarily end it elsewhere.Source: §2.1 · pp. 2–3

2.2 KV Reuse under Dynamic Agent Execution

  • The 90th-percentile longer-to-shorter ratios for adjacent non-LLM waits are 144.52× for Agent-G, 33.67× for Codex, and 204.43× for Agent-X.These intervals are shaped by tools and other runtime factors, with little temporal continuity in representative requests.Source: Figure 3 · p. 3, §2.2 · p. 3

2.3 Low KV Hit Rates under Semantic-Blind Policies

  • Realized KV hit ratios average 28.1% under LMC and 39.8% under MORI, below the 82.8% average theoretical maximum; prefill accounts for 37.6% and 34.6% of execution time, respectively.The maximum is measured by offline replay; prefill shares are averaged across the three workloads.Source: Figure 4 · p. 4, §2.3 · p. 4

2.4 Two Failures Caused by the Semantic Gap

  • LMC zombie-page shares peak at 31.1%, 36.3%, and 15.8% on Agent-X, Codex, and Agent-G; MORI peaks at 49.7%, 33.1%, and 27.4%, respectively.Zombie KV has lost all future reuse value, yet can remain resident because the baselines do not track lifecycle ends.Source: Figure 5 · p. 4, §2.4 · p. 4
  • Wrong-direction eviction ratios are 79.5%, 95.0%, and 71.0% for LMC, versus 57.8%, 75.6%, and 45.7% for MORI on Agent-X, Codex, and Agent-G.Newly produced KV may wait through decoding, tools, sub-agents, or scheduling, so recency and observed idleness can misorder future reuse.Source: Figure 6 · p. 4, §2.4 · p. 4

2.5 Related Work

  • Prior KV substrates manage physical tokens, pages, and prefixes, while workflow-based policies assume predefined structure and prediction-based policies estimate reuse from timing signals.ReAct branches emerge incrementally, and unstable reuse intervals weaken prediction reliability.Source: §2.5 · p. 5

3 Challenges and Design Overview

KVTether addresses diverse harnesses, cross-layer KV complexity, and uncertain reuse intervals with a semantic primitive interface, an event-driven lifecycle manager, and state-prioritized cache control.

C1: Diverse agent harnesses

  • KVTether exports runtime lifecycle semantics through message-level primitives, giving heterogeneous harnesses a common interface without tying integration to harness-specific functions.The primitive model covers message edits, context creation, and lifetime declarations while preserving existing agent logic.Source: §3 · p. 5

C2: Cross-layer complexity

  • An event-driven lifecycle manager translates message events into KV lifecycle states while encapsulating tokenization, prefix invalidation, and shared-KV details.This keeps physical cache-management complexity out of agent harnesses.Source: §3 · pp. 5–6

C3: Uncertain reuse intervals

  • The cache engine bypasses or reclaims dead KV first, then orders live-but-idle KV using observed overall reuse trends rather than precise per-item prediction.This confines reuse uncertainty to the live-but-idle tier.Source: §3 · p. 6

4 Semantic Primitive Model

KVTether exposes context-management intent through message-level primitives compatible with existing harnesses, generating semantic events for lifecycle management. Its interface builds on shared message-sequence context managers and supports composition without exposing token or KV-page handling to the harness.

4.1 Message-Level Primitives

  • All four studied harnesses use context-management logic that maintains ordered messages and assembles LLM inputs, enabling lifecycle tracking at message granularity.Tracking updates captures context changes; explicit lifetime declarations capture retention intent.Source: §4.1 · p. 6
  • Eight primitives record message edits, context creation and inheritance, and message lifetime declarations, covering spatial context changes and temporal retention requirements.Combining primitives expresses more complex context operations and reuse lifetimes.Source: §4.1 · p. 6, Table 3 · p. 6

4.2 Integration and Composition

  • A Python adapter reports existing context-manager operations through the SDK, while compaction marks its temporary context one-off and replaces selected history with a summary.The harness continues composing prompts without handling tokens or KV pages; System messages remain outside the replaced range, and later rounds reuse the summary.Source: §4.2 · pp. 6–7, Figure 8 · p. 7

5 Event-Driven Lifecycle Management

KVTether’s event-driven manager resolves message-level changes into KV scope and lifecycle states, accounting for prefix validity, materialized KV, and independent context retention. Those states distinguish active use, suspended reuse, expired lifetimes, and temporary single-inference messages.

5.1 KV Scope Resolution

  • The lifecycle manager translates semantic events into KV actions by resolving their scope against message-to-KV mappings, prefix validity, and sharing across contexts.Context-scoped events update a context directly, while message-scoped events require translation.Source: §5 · p. 7, §5.1 · p. 7, Figure 9 · p. 7
  • Middle-message removal or replacement invalidates the cached suffix because KV validity depends on the preceding token prefix.Only messages materialized through inference have token and KV ranges to manage; updates are applied at inference boundaries.Source: §5.1 · p. 7

5.2 Lifecycle State Management

  • KVTether tracks each message version per context as pinned, active, suspended, expired, or temporary, updating states as requests run and contexts change.Forked contexts keep independent states; termination expires ordinary messages, while pinned messages retain protection and one-off contexts mark messages temporary.Source: §5.2 · pp. 7–8

6 State-Guided KV Cache Engine

KVTether first uses lifecycle states to resolve cache decisions whose outcomes are semantically determined, then orders live-but-idle KV under pressure. Its MRU heuristic preserves suspended KV longer when reuse tends to occur after a delay.

6.1 Separating Determined and Uncertain Reuse

  • KVTether filters determined cases before uncertain reuse: Temporary KV bypasses admission, Expired KV is reclaimed first, and Suspended KV yields before Active or Pinned KV.This ordering reduces the chance that dead KV or higher-priority live KV distorts competition within the uncertain idle tier.Source: §6.1 · p. 8
  • Shared KV takes the highest-priority state among mapped message versions and becomes Discarded only after all versions are Discarded and in-flight use completes.A forked child’s Suspended reference, for example, keeps shared KV Suspended after its parent removes the message.Source: §6.1 · p. 8

6.2 Ordering Suspended-KV Under Pressure

  • After state filtering, the remaining eviction-order uncertainty is concentrated among live, currently idle KV in the Suspended tier.LRU assumes the most recently accessed entry is closest to reuse, an assumption misaligned with suspended KV waiting through tools, subagents, or other physical stages.Source: §6.2 · p. 8, §6.1 · p. 8
  • Across three workloads, observed returns cluster seconds after writeback: 43.1% fall within 1–3 s for Agent-G, 46.6% within 3–6 s for Codex, and 87.5% within 2–4 s for Agent-X.KVTether therefore evicts the most recently suspended KV first, favoring entries that have waited longer and are more likely to be reused.Source: §6.2 · pp. 8–9
  • At high load on Agent-X, LMCache’s hit ratio falls to 10.2%, versus 49.5% for KVTether, as LRU can evict older suspended KV closer to reuse.The observed reuse intervals cluster several seconds after writeback, rather than at the youngest ages.Source: §6.2 · pp. 8–9, Figure 10 · p. 9

7 Implementation of KVTether

KVTether exports ordered lifecycle events from harness primitives to a server-side lifecycle manager, which coordinates cache actions with the serving stack. Its state-guided cache engine separates global lifecycle control from per-tier caching.

Implementation of KVTether

  • The in-harness framework records primitive calls synchronously and exports them asynchronously in order, keeping lifecycle export off the critical path without changing event order.This component exposes eight primitives.Source: §7 · p. 9
  • A server-side Python lifecycle manager ingests events, reproduces prompt formatting and tokenization to map messages to KV, and coordinates cache actions before forwarding LLM calls.It passes the call ID and context ID to the cache engine so admission and bypass decisions target the correct context-owned KV ranges.Source: §7 · p. 9
  • The multi-tier cache engine separates global decisions about reusable KV from per-tier decisions about retaining fast copies, using a native CUDA and MoonCake engine or compatibility proxies for existing backends.The cache tiers span GPU HBM, host memory, and remote memory.Source: §7 · p. 9

8 Evaluation

Across Agent-X, Agent-G, and Codex, KVTether improves cache reuse and reduces prefill, latency, and estimated task pricing against LMCache and MORI. Ablations show separate contributions from lifecycle tracking and recency optimization; integration and runtime overhead are also evaluated.

8.1 Experimental Setup

  • The evaluation replays three workloads on a 12-GPU, three-node cluster, preserving lifecycle-event order and using identical task samples and arrival sequences across methods.Baselines are LMCache, which uses page-access LRU, and MORI, which ranks programs by recent execution time spent in tool calls.Source: §8.1 · pp. 9–10

8.2 End-to-End Performance

  • KVTether reduces average end-to-end latency by 16.5% versus LMCache and 12.4% versus MORI, while lowering average estimated task pricing by 40.0% and 33.2%, respectively.Maximum latency reductions are 26.3% versus LMCache and 17.4% versus MORI; maximum task-pricing reductions are 53.6% and 44.1%, respectively.Source: §8.2 · p. 10

8.3 Ablation Study

  • Against KVTether without lifecycle tracking, KVTether gains 25.6% average absolute hit ratio and reduces average prefill time, latency, and task pricing by 23.2%, 13.1%, and 33.1%, respectively.Without lifecycle tracking, zombie pages average 22.1%, 21.4%, and 38.5% of resident KV on Codex, Agent-X, and Agent-G; KVTether has below 2.5% on Codex and none on the other workloads.Source: §8.3.1 · p. 11, Figure 13 · p. 11
  • Against KVTether without recency optimization, KVTether gains 29.1% average absolute hit ratio and reduces average prefill time, latency, and task pricing by 26.1%, 14.8%, and 35.4%, respectively.Wrong-direction evictions under the variant are 65.6%, 75.0%, and 92.3% on Agent-X, Agent-G, and Codex; KVTether reduces them to 38.7%, 25.7%, and 38.6%.Source: Figure 14 · p. 11, §8.3.2 · pp. 11–12

8.4 Harness Integration

  • Integration into six harnesses takes 26–216 lines per harness, including 14–162 lines for the reusable context manager and 12–54 lines for function integration.In the Codex examples, the wrapper adds 11 lines, while the safety guardian and subagent-call paths each add four lines.Source: Table 5 · p. 12, §8.4 · p. 12, Figure 15 · p. 12

8.5 Runtime Overhead

  • Runtime overhead is estimated at 2.0% of average LLM-call duration for 256K-token inputs, with primitive recording under 100 μs per invocation and call interposition under 50 ms.The lifecycle runtime and cache engine control path each use one CPU core and less than 2 GB of host memory across the experiments.Source: §8.5 · pp. 12–13

9 Conclusion

KVTether bridges message-level context management and physical KV storage with lifecycle primitives, an event-driven manager, and state-guided cache control. It reduces estimated task pricing and end-to-end latency against LMCache and MORI, with integration into six harnesses requiring at most 216 lines per harness.

9 Conclusion

  • KVTether translates message-level lifecycle semantics into KV states, then bypasses or reclaims Discarded KV and applies MRU within the live-idle tier.Across evaluations, average task pricing falls by 40.0% and 33.2%, and end-to-end latency by up to 26.3% and 17.4%, relative to LMCache and MORI, respectively.Source: §9 · p. 13
Loading 2609.39819v1…