PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentServeSim: Serving-System Simulation and Policy Search for LLM Agent Programs

Rakibul Hasan Rajib, Mengxin Zheng, Qian Lou

arXiv:2606.09613v3cs.CLcs.AI

TL;DR

Agent-serving evaluation is costly, and a program’s completion time depends on KV retention, routing, and scheduling across tool gaps and model turns. AgentServeSim models programs and their policy-dependent execution, with mean-JCT error within 5.5% on B200 and 5.2% in the saturated RTX PRO 6000 regime, and supports LLM-driven policy search.

  • Problem

    Evaluating agent-serving policies is costly, while job completion time depends on program-wide KV retention, routing, and scheduling across tool gaps and model turns.

    Source: §1 · pp. 1–2

  • Method

    AgentServeSim simulates agent programs, maintaining cross-turn state and releasing successor turns from simulated predecessor completions while exposing retention, routing, and scheduling decisions.

    Source: §1 · p. 2

  • Results

    Mean-JCT error remains within 5.5% on B200 and 5.2% in the saturated RTX PRO 6000 regime; searched policies improve mean JCT over hand-written seeds by 0.5% for retention and 2.8% for scheduling.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AgentServeSim serves as a CPU-based evaluator for LLM-driven policy search and controlled exploration of interacting agent-serving policies.

    Source: §5.3 · p. 7, §6 · p. 8

Abstract

from arXiv · show

Large language model agents execute programs comprising multiple model turns interleaved with external tool calls. Their job completion time depends on how the serving system retains KV state across tool gaps, routes successor turns, and schedules competing programs. Most existing serving simulators operate on request streams in which arrivals are externally supplied and KV state follows request- or cache-scoped semantics. They therefore cannot jointly represent the cross-turn state and policy-dependent successor releases needed to evaluate counterfactual agent-serving trajectories. We present AgentServeSim, a simulator whose unit of execution is the agent program. A Program Control Block maintains cross-turn state, while a Program Orchestrator causally releases successor turns from simulated predecessor completions. A Retention Plane controls KV state across tool gaps, and a Dispatch Plane determines where and when each ready turn executes. We validate AgentServeSim against real vLLM deployments in 20 paired simulator-real cells spanning two GPU platforms, Llama-3.1-8B and Llama-3.1-70B, coding and function-calling agents, and five arrival rates. Mean JCT error remains within 5.5% on B200 and 5.2% in the saturated RTX PRO 6000 regime. Finally, we propose LLM-driven automated agent-serving policy search using AgentServeSim as a CPU-based fitness evaluator. The resulting policies improve mean JCT over hand-written seed policies by 0.5% for KV retention and 2.8% for scheduling.

1 Introduction

Agent-serving evaluation must account for program-wide KV state and successor turns released by policy-dependent execution, rather than treating calls as independent requests. AgentServeSim models those dependencies and supports policy search, with validation against real vLLM deployments.

1 Introduction

  • Existing simulators lack a general program-level abstraction, while repeated real-engine evaluation is costly across interacting retention, routing, and scheduling choices.Agent programs retain state across calls, unlike models centered on request arrivals and request-scoped KV lifetimes.Source: §1 · p. 2
  • AgentServeSim maintains cross-turn program state and causally releases successor turns, while its Retention and Dispatch Planes control KV management, routing, and scheduling.The simulator generates policy-dependent release, placement, and cache-residency trajectories for a fixed logical program.Source: §1 · p. 2
  • Mean-JCT error remains within 5.5% on B200 and 5.2% in the saturated RTX PRO 6000 regime; searched policies improve mean JCT over hand-written seeds by 0.5% for KV retention and 2.8% for scheduling.Validation spans 20 paired simulator–real cells across two GPU platforms, two model scales, two agent workloads, and five arrival rates.Source: §1 · p. 2
  • LLM-driven policy search uses AgentServeSim as a CPU-based evaluator to discover retention and scheduling policies that outperform hand-written seed policies.The paper presents this as an alternative to evaluating every policy by repeatedly deploying it in a real serving engine.Source: §1 · pp. 2–3

2 Related Work

Prior serving simulators model request streams or stateful request chains, while agent-serving systems evaluate particular policies in production engines. AgentServeSim targets joint program-level evaluation of retention, routing, and scheduling.

Serving simulators

  • Request-stream simulators model independent requests, while Frontier releases stateful successors causally but pins them to the predecessor’s replica and relies on native prefix caching.Revati reduces simulator–engine policy drift through engine emulation; AgentServeSim instead compares decisions from the same program-level policy in simulation and real execution.Source: §2 · p. 3

Agent-serving systems

  • Agent-serving systems demonstrate program-level policies, but production-engine evaluation makes cross-policy counterfactuals expensive to isolate.AgentServeSim separates retention, routing, and scheduling behind a common program-state abstraction so components can be composed and compared in simulation.Source: §2 · p. 3

3 AgentServeSim Design

AgentServeSim models each program across calls, with shared policy interfaces for dispatch and retention. Its orchestrator generates successor releases from simulated execution, while the two planes make program-aware serving decisions.

3.1 System Overview

  • The System Simulator evaluates each dispatched batch, while the Retention Plane assigns KV value and a protection deadline after call completion and the Orchestrator releases the next call after its tool duration.The system reports metrics including memory use, energy, throughput, JCT, preemptions, and per-call latency measures.Source: §3.1 · pp. 3–4

3.2 Program Orchestrator

  • The Program Orchestrator maintains a PCB for each live program and releases successor calls at predecessor completion plus tool duration, generating policy-dependent release times.PCB state includes program position, attained service, prior placement, KV residency and protection, and tool state; future trace information is withheld until revealed by execution.Source: §3.2 · p. 4

3.3 Dispatch Plane

  • The Dispatch Plane selects an instance and scheduling priority using persistent program state; Program-FCFS preserves program arrival order, while PLAS favors programs with less attained service.Routing is evaluated first because the selected instance determines the destination queue and whether the program prefix is resident.Source: §3.3 · pp. 4–5

3.4 Retention Plane

  • The Retention Plane assigns KV value and a protection deadline, then reclaims entries by expiry and value under memory pressure; its policies make cross-turn KV lifetime programmable.Evaluated deadlines include immediate release, fixed TTL, per-tool TTL, and pressure-aware TTL; protection is a preference rather than a lock.Source: §3.4 · p. 5

4 Experimental Setup

The validation setup pairs AgentServeSim with vLLM across hardware, model, workload, and arrival-rate configurations. The simulator uses device- and model-specific profiles and replays captured call inputs and tool durations.

Hardware platforms

  • Validation uses B200 GPUs with Llama-3.1-8B at TP1 and Llama-3.1-70B at TP2, plus RTX PRO 6000 GPUs with Llama-3.1-70B at TP2.Each cell uses a single serving instance, so validation exercises retention and scheduling but not routing.Source: §4 · p. 6

Simulator configuration

  • The simulator matches device specifications, profiles operator latency per device–model pair, models collectives with measured all-reduce bandwidth, and takes KV-pool size from the engine’s allocation report.A measured residual table covers a grid of batch shapes.Source: §4 · p. 6

Workloads

  • The workloads comprise 50 SWE-bench programs and 100 BFCL v4 programs, submitted under Poisson arrivals from 0.02 to 0.1 jobs per second.Both sides replay captured prompts, generated-token counts, and tool durations, while release times, ordering, and residency are generated during execution.Source: §4 · p. 6

5 Results

AgentServeSim closely matches vLLM on program-aware serving, and its policy sweeps show that retention and scheduling interact. LLM-driven search finds mean-JCT improvements over hand-written seeds on both policy axes.

5.1 Fidelity under program-aware serving policies

  • p99 JCT error stays within ±2.7% on B200 and reaches +7.1% on RTX PRO 6000; per-program MAPE remains within 5.9% and 11.2%, respectively.The MAPE bounds are 5.9% on B200 and 11.2% on RTX PRO 6000.Source: §5.1 · p. 7
  • Across 20 paired cells, mean-JCT error stays within ±3.5% on B200 SWE-bench and ranges from −5.2% to +4.5% on RTX PRO 6000.The RTX PRO 6000 regime reproduces memory pressure across all five arrival rates.Source: §5.1 · pp. 6–7

5.2 Agent-serving design-space exploration

  • At the RTX memory-pressure point, pressure-aware TTL with PLAS reduces JCT by 5.0%, while changing retention alone under FCFS increases it by 0.4%.Switching from FCFS to PLAS under LRU reduces JCT by 3.9%; each simulation point costs about one CPU-core-hour versus two GPUs for three to six hours for vLLM.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 LLM-driven evolutionary policy search

  • The search loop has an LLM modify executable policy code, evaluates each candidate on a complete agent workload, and uses measured JCT to guide subsequent modifications.Candidates use deployment-available state rather than future trace information; the search ran 50 iterations per policy axis.Source: §5.3 · pp. 7–8
  • The evolved retention policy reduces mean JCT by 0.5% relative to the TTL seed, estimating tool gaps and adapting protection to context length and KV-pool utilization.It uses an exponential moving average of each tool's gap estimate.Source: §5.3 · p. 8, Table 2 · p. 8
  • The evolved scheduling policy reduces mean JCT by 2.8% on the target cell and by 2.3% at 0.02 JPS, using context length and KV residency in its priority score.The policy replaces attained service with its square root; lower scores run first.Source: §5.3 · p. 8, Table 2 · p. 8

6 Conclusion

AgentServeSim models agent serving at the program level and reproduces real-engine mean JCT within stated bounds across paired cells. It also supports policy search that improves mean JCT over hand-written seed policies.

  • Mean-JCT error stays within 5.5% on B200 and 5.2% on RTX PRO 6000 across 20 paired simulator–real cells, including under memory pressure.Source: §6 · p. 8
  • Program Control Blocks and a Program Orchestrator maintain execution across calls, while Retention and Dispatch Planes expose KV-management, routing, and scheduling decisions through a common vLLM interface.Source: §6 · p. 8
  • LLM-driven evolutionary policy search uses AgentServeSim as a CPU-based evaluator and produces retention and scheduling policies that improve mean JCT over hand-written seeds.Source: §6 · p. 8

A Workloads

The workloads use recorded traces from coding and function-calling agents, with per-turn request shapes and tool gaps replayed under Poisson program arrivals.

A Workloads

  • Traces came from mini-SWE-agent on SWE-bench instances and a multi-turn harness on BFCL v4, with each turn recording prompt and output token counts, tool identity, and tool duration.Source: A Workloads · p. 10
  • Table 3 reports trace statistics, with context tokens measured per turn because each turn’s prompt contains its predecessors’ context.Source: Table 3 · p. 10
  • Programs arrive according to Poisson schedules of 0.02, 0.04, 0.06, 0.08, and 0.1 JPS; the SWE-bench trace totals 38.7M context tokens.Source: A Workloads · p. 10

B Per-cell results

The per-cell results report simulator-versus-real JCT and per-program errors under TTL retention and PLAS scheduling, while request-level simulation compares several latency and throughput metrics against real vLLM.

B Per-cell results

  • Per-cell results cover mean-JCT error, p99-JCT error, per-program MAPE, and p90 absolute error under TTL retention and PLAS scheduling with τ = 2 s.Source: B Per-cell results · p. 10

C Request-level Simulation

D TTFT, TPOT and Throughput Validation

Per-call validation reports close TTFT and TPOT values for one B200 SWE-bench cell and tracks instantaneous throughput against the real trace. The supplied figure caption identifies per-arrival-rate p99-JCT error and per-program MAPE comparisons.

D TTFT, TPOT and Throughput Validation

E Policy interfaces

Retention, scheduling, and routing policies return declarative decisions, while the shipped policy set covers KV retention across tool gaps and dispatch choices. The section also compares simulator and real-serving timing and throughput in a specified B200 workload.

E Policy interfaces

  • Retention and scheduling callbacks return protect, release, evict, or none decisions and an integer priority, with smaller priorities scheduled first.Source: E Policy interfaces · p. 11
  • Figure 4 compares mean TTFT and TPOT for real vLLM and AgentServeSim; Figure 5 compares their instantaneous generated-token throughput in 10-second bins.The timing comparison covers 2,873 calls under TTL retention and PLAS scheduling on B200 8B SWE-bench at 0.04 JPS; throughput uses per-call token counts and first- and last-token times.Source: Figure 4 · p. 12, Figure 5 · p. 12
  • Shipped policies include six KV-retention choices, three scheduling choices, and three routing choices.Retention includes fixed and pressure-scaled TTL, Continuum per-tool TTL, and min-waste; scheduling includes FCFS, program-FCFS, and PLAS; routing includes round-robin, least-loaded, and session affinity.Source: E Policy interfaces · p. 12
  • Pressure-scaled TTL shortens its deadline linearly from τ at 70% KV utilization to τ/2 at 90%, while min-waste compares expected-gap recomputation cost with eviction waste.Min-waste retains when expected gap times context length is no greater than the token-seconds wasted by eviction and re-prefill.Source: E Policy interfaces · p. 12

F Seed and evolved policies

The section contrasts hand-written TTL and PLAS seeds with evolved retention and scheduling policies, then reports how policy search selected and refined candidates. Both seeds and evolved policies use observable program state, time, and KV-utilization signals.

F Seed and evolved policies

  • Of 48 retention candidates, 40 were within 0.5% of one another; scheduling search found that context-length and residency terms accounted for most of the gain.Search ran for 50 iterations with a population of 40; the square-root term appeared in the final iteration.Source: F Seed and evolved policies · p. 12
  • Both seed and evolved policies read only PCB fields, current time, and the KV-utilization signal.Source: F Seed and evolved policies · p. 12
  • The hand-written retention seed is TTL with τ = 2 s, and the hand-written scheduling seed is PLAS.Source: F Seed and evolved policies · pp. 12–13
  • The evolved retention code tracks observed tool gaps with an EMA and scales its estimated gap by context length before setting a protection deadline.The shown scale is max(0.5, context_tokens / 20000.0), and the gap EMA updates as 0.8 times its prior value plus 0.2 times the observed gap.Source: F Seed and evolved policies · pp. 12–13
  • The evolved scheduler combines square-root attained service with a context-length penalty and a residency bonus, using the shown per-token weights of 0.03 ms and 0.10 ms.Resident KV lowers priority values, while larger contexts increase them; smaller scheduling priorities run first.Source: F Seed and evolved policies · p. 13
Loading 2606.09613v3…