PaperlayerView exact revision on arXiv

Source-linked AI summary

CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents

Shubham Tiwari, Tapan Chugh, Nash Rickert, Simon Peter, Ratul Mahajan, Haiying Shen

arXiv:2606.16824v1cs.DCcs.OS

TL;DR

Coding-agent serving had not been studied extensively, with a lack of real-world traces identified as a reason for the gap. CacheWise combines prefix-aware scheduling with predictive eviction guided by tool-execution metadata. On real coding-agent traces, it reduces KVCache evictions by up to 2–2.6× and improves end-to-end session completion time by up to ~3.5×.

  • Problem

    Coding-agent serving had not been examined extensively, in part because publicly available real-world workload traces were lacking.

    Source: §1 · p. 1

  • Method

    CacheWise combines prefix-aware request scheduling with reuse-aware KVCache eviction predicted from session metadata such as tool execution information.

    Source: §1 · p. 2

  • Results

    CacheWise reduces KVCache evictions by up to 2–2.6× and improves end-to-end session completion time by up to ~3.5× on real coding-agent traces.

    Source: §8 · p. 12

  • Takeaways & Limitations

    The trace analysis characterizes coding agents as a distinct LLM serving workload with closed-loop execution, long-lived sessions, and growing prefixes.

    Source: §1 · p. 2

    The evaluation trains the predictor on 80% of sessions and uses the remainder for experiments; robustness to trace-distribution drift is left for future study.

    Source: §6.1 · p. 8

Abstract

from arXiv · show

Coding agents are a fast-growing LLM application, executing as long-running closed-loop sessions in which LLM generations alternate with external tool calls. Yet, unlike chat workloads, their serving behavior has not been studied extensively. We address this gap by collecting a dataset of real-world coding assistant traces. Our analysis shows that coding agent sessions repeatedly reuse large prefixes and create sustained KVCache pressure that conventional LLM serving policies handle poorly. Based on our analysis, we present CacheWise, a KVCache management layer that improves KVCache reuse for coding agent workloads. CacheWise combines prefix-aware scheduling with reuse-aware eviction guided by lightweight predictions from tool call metadata. Implemented in vLLM and evaluated on the collected traces, CacheWise reduces KVCache evictions by up to 2-2.6x and improves total agent session completion time by up to 3.5x.

1 Introduction

Coding-agent serving had not been examined extensively, in part because real-world traces were unavailable. CacheWise addresses this gap with a trace-based workload study and a KVCache management layer combining prefix-aware scheduling with reuse-aware eviction.

1 Introduction

  • Unlike chat serving, which has been studied extensively [3] [14] [21] [26] [35] [45] [46], coding-agent sessions alternate LLM requests and tool calls, reuse and extend prefixes, and are evaluated by full-session completion time.Tool-initiated turns dominate these long-running sessions relative to direct user input.Source: §1 · p. 1
  • Existing serving policies such as FCFS batching and LRU eviction ignore coding-agent prefix overlap and tool-execution timing, risking KVCache thrashing and lower token goodput under memory pressure.Prior agent-serving approaches [15] [23] tightly couple agent frameworks with serving systems, limiting their suitability for large-scale deployments with diverse clients.Source: §1 · p. 2
  • CacheWise prioritizes requests with resident-prefix overlap and predicts prefix reuse from session metadata, including tool execution information, without modifying coding agents.Lightweight predictors use historical samples collected by the serving system.Source: §1 · p. 2
  • The collected traces characterize coding agents as a distinct serving workload with closed-loop execution, long-lived sessions, and large, growing prefixes.The dataset includes real-world coding-assistant traces from researchers actively using coding assistants for research.Source: §1 · pp. 1–2
  • CacheWise reduces KVCache evictions by 2–2.6× compared with systems such as vLLM and improves total request completion time by up to ~3.5×, approaching predictive eviction with ground-truth reuse information.The evaluation uses real-world coding-agent traces.Source: §1 · p. 2

2 Background: Efficient LLM Serving

LLM serving uses scarce accelerator memory for KVCache states, which constrains both prefill and decode efficiency. When requests share prefixes, memory pressure can force eviction and later recomputation or transfer of cached state.

2 Background: Efficient LLM Serving

  • Efficient XPU memory management is crucial across both prefill and decode, despite their different optimization needs.Serving systems use massively parallel operations to exploit accelerator compute and memory resources.Source: §2 · p. 2
  • Accelerator memory for KVCache states limits the efficiency benefits of parallel prefill and batched decode.Prefill computes key-value tensors for input tokens, while decode generates output tokens sequentially.Source: §2 · p. 2
  • During prefill, serving systems may persist shared-prefix KVCache, but memory pressure can force eviction and require later recomputation or transfer from lower storage tiers.State-of-the-art systems commonly manage KVCache in fixed-size blocks and make eviction decisions per block.Source: §2 · p. 2

3 Coding Agent Workloads

CATraces provides detailed real-world coding-assistant sessions whose many turns, large growing contexts, and tool-driven requests distinguish them from chatbot workloads. Long sessions and variable tool times create sustained KVCache pressure and irregular reuse intervals.

3 Coding Agent Workloads

  • CATraces contains detailed Claude Code conversations, tool results, and system metadata, providing real-world coding-assistant traces where prior datasets were unavailable.The traces were collected from researchers using coding assistants for software development and engineering tasks on open-source projects.Source: §3 · p. 3
  • Coding-agent requests have ~21× higher prefill-to-decode token ratios than chatbot requests, and their sessions contain orders of magnitude more turns.Each turn offers a reuse opportunity; recomputing the accumulated prefix at every turn would require orders of magnitude more FLOPs.Source: §3 · p. 3, Figure 2 · p. 4, Figure 3 · p. 4
  • Tool-completion requests are 20× more frequent than user-initiated requests at the median, making request generation predominantly closed-loop.Session completion time captures multi-step execution efficiency, while TTFT and TBT measure per-request latency.Source: §3 · p. 3, Figure 4 · p. 4

Long-running sessions compete for limited accelerator

4 Implications for Efficient LLM Serving

Coding-agent requests within a session repeatedly reuse substantial KVCache prefixes, but request-level allocation and eviction policies overlook future reuse. Interleaving sessions and relying on recency can therefore cause repeated eviction and restoration of state.

4 Implications for Efficient LLM Serving

  • Successive requests within coding-agent sessions share substantial KVCache state, while request-level policies ignore future reuse and reduce cache reuse and system efficiency.The active working set can exceed accelerator memory as sessions accumulate context.Source: §4 · pp. 4–5
  • FCFS interleaves queued sessions, expanding the working set and evicting state that sessions later reuse, which causes repeated eviction and recomputation or movement.This behavior is described as KVCache thrashing.Source: §4 · p. 5
  • The ideal eviction choice is the session whose resident blocks have the longest time until next reuse, preserving blocks needed soonest.Evicting a session with a short time to reuse forces an imminent recomputation or data-movement penalty.Source: §4 · p. 5
  • Because future reuse time is unknown, LRU can evict high-reuse-value blocks that must then be recomputed or moved back into memory.Repeated eviction and restoration adds recomputation and data-movement overhead.Source: §4 · p. 5

5 CACHEWISE Design

CACHEWISE manages KVCache for long-running, closed-loop sessions using prefix-aware scheduling and predictive eviction. It prioritizes requests that reuse resident prefixes and estimates relative reuse times from tool-call metadata to guide eviction.

5 CACHEWISE Design

  • CACHEWISE targets reuse across requests in long-running sessions, combining prefix-aware scheduling with predictive KVCache eviction.The design is intended to reduce eviction overhead and support concurrent sessions and session performance.Source: §5 · pp. 5–6, Table 2 · p. 6

5.1 Prefix-aware Request Scheduling

  • Prefix-aware scheduling dispatches the request needing the fewest additional KVCache blocks, prioritizing overlap with state already resident in accelerator memory.For closed-loop sessions, this approximates shortest-job-first scheduling and favors session completion over per-request responsiveness.Source: §5.1 · p. 6, Figure 11 · p. 7

5.2 Predictive KVCache Eviction

  • Predictive eviction approximates the optimal policy by evicting blocks in decreasing order of predicted time until reuse; only the relative ordering across sessions is needed.Lightweight estimates based on tool-call names and arguments can identify the session with the longest reuse interval.Source: §5.2 · p. 6
  • Tool-name-only duration distributions can be insufficient, so CACHEWISE clusters historical tool arguments using TF-IDF and uses per-cluster duration distributions.Tool execution time can vary with arguments and environment, including the repository’s test suite.Source: §5.2 · p. 7, Figure 12 · p. 7

5.3 Implementation

  • The vLLM implementation attaches tool metadata to KVCache blocks, prioritizes unreferenced blocks in an eviction heap, and periodically refreshes predictions.The rebuild interval trades fresher estimates against CPU scheduling overhead; the evaluation uses N_rebuild = 3.Source: §5.3 · pp. 7–8

6 Evaluation

CACHEWISE is evaluated by replaying real coding-agent traces under increasing concurrent-session loads. It improves session completion time and KVCache reuse, with benefits from both predictive eviction and prefix-aware scheduling.

6.2 End-to-End Performance Improvements

  • At N>10, CACHEWISE achieves 2.7×–3.5× lower session completion time than vLLM and InferCept, comparable to ground-truth CACHEWISE*.The comparison uses Figure 13; performance is similar across systems at N≤10.Source: Figure 13 · p. 8, §6.2 · p. 9

6.1 Experimental Setup

  • Evaluation deterministically replays dataset traces using Qwen2.5-Coder-32B-Instruct, launching N independent coding-agent sessions concurrently on one serving instance under increasing load and measuring token goodput, request and session completion times, and KVCache transfer volume.They randomly sample 80% of sessions for offline predictor training and use the remaining traces for experiments.Source: §6.1 · pp. 8–9

6.3 Performance Ablations

  • Against prefix-aware vLLM and InferCept, predictive eviction adds 1.2×–1.6× higher token goodput and ~1.7×–2× lower session completion time.The comparison holds prefix-aware scheduling fixed.Source: §6.3 · p. 9, Figure 15 · p. 10
  • Prefix-aware scheduling alone improves token goodput ~1.38×–1.64× at N=30 and ~1.6×–1.7× at N=40, while reducing session completion time ~1.8×–2.35× and ~1.85×–2.66×, respectively.The gains persist without accurate KVCache reuse predictions.Source: §6.3 · p. 9, Figure 16 · p. 10

6.4 KVCache Movement

  • CACHEWISE cuts KVCache transfer volume ~2×–2.6× and, with physical GPU–CPU offloading, yields ~1.19× lower session completion time than vLLM at N=30.Under offloading, the absolute gains are smaller because PCIe transfers are faster than prefill recomputation for large prefixes.Source: Figure 17 · p. 11, §6.4 · p. 11, Figure 18 · p. 11

6.5 Predictor Accuracy

  • Refining tool-duration prediction from Point Estimate to C100 lowers evictions and session completion time across N=30, 40, and 50; C100 improves session completion time by up to 19%.Increasing cluster count beyond 100 yielded no additional clusters because the data did not support finer granularity.Source: Figure 19 · p. 11, §6.5 · p. 11

6.6 Scheduling Overheads

  • At N=40, CPU scheduling rose from 0.33 s to 0.99 s, while GPU model execution fell from 16.6 s to 11.2 s (1.48×), yielding a net reduction of ~4.7 s per request.The added scheduling overhead is offset by lower model execution time.Source: Figure 20 · p. 12, §6.6 · p. 12

7 Related Work

Related work addresses agent-serving abstractions, request scheduling, predictive eviction, and tiered KVCache storage. CACHEWISE focuses on adapting the serving system to coding-agent session characteristics.

Serving systems for emerging LLM applications

  • Pie and Autellix introduce programming abstractions for agentic workloads, whereas CACHEWISE modifies the serving system and can work with existing widely deployed coding agents.Source: §7 · p. 12

Session-aware vs. request-aware scheduling

  • Unlike request-oriented scheduling and system-prompt sharing, CACHEWISE targets prefix reuse across long-running, closed-loop coding-agent sessions.vLLM and InferCept schedule requests independently, while SGLang's prefix-aware scheduling primarily targets sharing across independent requests.Source: §7 · p. 12

Predictive vs. recency-based eviction

  • CACHEWISE predicts KVCache reuse using session metadata, while prior systems use recency, tool-duration moving averages, speculative offloading, or theoretical predictive caching.Reducing eviction frequency is complementary to tiered storage because it reduces the data those systems must move across tiers.Source: §7 · p. 12

8 Conclusion

Coding agents are long-running closed loops with tool-initiated turns and repeatedly reused, growing prefixes. CACHEWISE combines prefix-aware scheduling with predictive eviction, reducing evictions and session completion time on real coding-agent traces.

8 Conclusion

  • Coding agents differ from traditional chat workloads through long-running closed loops, predominantly tool-initiated turns, and repeated reuse of large, growing prefixes.Serving efficiency depends on preserving and reusing KVCache across turns within a session.Source: §8 · p. 12
  • CACHEWISE prioritizes requests able to reuse resident KVCache and predicts which prefixes will be reused soon from tool execution metadata.Source: §8 · p. 12
  • On real coding-agent traces, CACHEWISE reduces KVCache evictions by up to 2–2.6× and improves end-to-end session completion time by up to ~3.5×.Source: §8 · p. 12

Related papers

Loading 2606.16824v1…