PaperlayerView exact revision on arXiv

Source-linked AI summary

Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systems

Wonung Kim, Hyunmin Choi, Minsu Kim, Jaehong Cho, Yeongwook Kim, Jongse Park

arXiv:2608.24650v1cs.ARcs.AI

TL;DR

LLM serving simulation must cover an expanding, costly design space, but rapid evolution and non-monolithic workflows strain human-maintained monolithic simulators. Simthesizer combines a composable unified dynamic-DAG simulator with a guarded coding agent that lowers feature requests into validated extensions. Its extensions achieve 2.51% average throughput error versus 6.03% for existing-simulator extensions and simulate up to 284.96× faster.

  • Problem

    Real-system evaluation is costly, while rapidly evolving and non-monolithic LLM serving exceeds the assumptions of manually maintained fixed simulation pipelines.

  • Method

    Simthesizer combines a composable simulator that uniformly represents complete serving workflows with a harnessed coding agent for guarded, validated extensions.

  • Results

    2.51% average throughput error versus 6.03% for existing-simulator extensions, with simulation up to 284.96× and 23.19× faster on identical workloads.

  • Takeaways & Limitations

    The results support pairing composable simulator infrastructure with agent-driven lowering as a scalable path for accurate and extensible LLM serving simulation.

Abstract

from arXiv · show

System-level simulation is an essential tool for exploring the rapidly expanding design space of LLM serving systems, where real deployments remain costly and often infeasible. However, modern LLM serving now evolves faster than human-driven simulator development can track, and emerging workloads and mechanisms, from agentic workflows to disaggregated serving, no longer fit the monolithic simulation pipeline that existing simulators assume. Each new mechanism therefore demands an invasive rewrite, leaving a widening development gap between deployed serving systems and the simulators that model them. To close this gap, we present Borg, a framework that realizes agent-driven simulator development. Borg introduces a composable simulator infrastructure that uniformly expresses the complete serving workflow, including the control decisions that coordinate it, and realizes it as a unified dynamic graph in Borg simulator. Synthesizer agent, a harnessed coding agent, then lowers natural-language feature requests onto this abstraction under simulator-specific guardrails and fidelity validation, evolving one shared simulator instead of building a new one for every feature. Under the same coding agent and harnesses, extensions built on Borg follow a vLLM-based real system with 2.51% average throughput error, versus 6.03% for extensions built on existing simulators. On identical workloads, Borg also simulates up to 284.96x and 23.19x faster than two state-of-the-art simulators, LLMServingSim2.0 and Vidur, respectively.

I. INTRODUCTION

System-level simulation is essential for exploring costly LLM serving designs, but fast evolution and non-monolithic workflows make existing simulators difficult to maintain. Simthesizer addresses this gap with composable infrastructure and agent-driven extension.

  • Real-system evaluation is often prohibitively expensive, making system-level simulation essential for exploring LLM serving design trade-offs.
  • Fast ecosystem evolution repeatedly invalidates simulator assumptions and requires manual implementation, integration, and validation effort.
  • Non-monolithic serving mechanisms force invasive restructuring of fixed simulation pipelines.
  • Simthesizer uniformly expresses complete serving workflows, including control decisions, in a composable DAG with modular components.
  • Synthesizer agent lowers natural-language feature specifications into executable simulation logic using structured interfaces, harnesses, and validation.
  • 2.51% average throughput error versus 6.03% for existing-simulator extensions, with up to 284.96× and 23.19× faster simulation.

II. BACKGROUND AND MOTIVATION

LLM serving simulation faces an expanding design space, while existing simulators encode behavior through human-built components integrated into fixed pipelines. Their design assumes slow evolution and monolithic serving behavior.

  • LLM serving design spans parallelism, scheduling, batching, KV-cache management, workload composition, and device configurations.
  • Existing simulators use human-implemented components integrated into fixed simulation pipelines.
  • Their approach relies on serving systems evolving slowly enough for manual engineering to keep pace.
  • Their second premise is that serving behavior can be represented by a monolithic simulation pipeline.

B. The Shifting Landscape of LLM Serving

Modern LLM serving is evolving rapidly while shifting toward non-monolithic workflows that orchestrate separately managed components. These changes expose structural limits in existing simulators and motivate composable, guarded agent-driven development.

  • Fast evolution: LLM workloads and serving mechanisms now evolve rapidly across every level of the serving stack.
  • Fast evolution: Agentic workloads have shifted serving from fixed inference flows toward programmable, application-defined workflows.
  • Non-monolithic serving: Disaggregated serving, speculative decoding, and KV-cache management orchestrate separately managed components instead of a self-contained inference lifecycle.
  • Implications for existing simulators: These trends widen the development gap because new behaviors outpace human implementation, integration, and validation, while non-monolithic serving breaks fixed pipelines.
  • Design principles: A uniform composable representation should capture stages, dependencies, and state transitions without restructuring the execution engine.
  • Design principles: Agent-driven implementation must combine bounded interfaces and simulator-specific guardrails with semantic validation.

III. OVERVIEW OF SIMTHESIZER

Simthesizer combines a unified dynamic-DAG simulator with a harnessed coding agent that maps feature requests to abstractions, extends implementations, and validates fidelity. Its two-layer architecture separates execution from evolving serving behavior.

  • Simthesizer comprises a unified extensible simulator and a harnessed coding agent for integrating new serving mechanisms.
  • The agent refines user requests, maps decisions to simulator abstractions, implements missing functionality, and validates behavior against measurements or reference evidence.
  • The execution layer realizes serving workflows as a unified dynamic DAG, while the control layer constructs and modifies graphs through modular policy components.
  • This separation provides a stable execution substrate while serving behavior evolves through localized extensions.

A. Unified Dynamic DAG Abstraction

Simthesizer simulator represents complete serving execution as a unified dynamic DAG, combining compute, communication, and logical control nodes whose dependencies and runtime behavior determine execution.

  • Execution graph representation: The unified dynamic DAG contains compute nodes, communication nodes, and logical nodes for policy events such as request arrival and scheduling.Edges encode execution-order dependencies between nodes.
  • Execution graph representation: Dynamic DAG modification APIs provide the interface for changing the execution graph during simulation.The supplied table is identified as the Dynamic DAG Modification APIs.
  • Execution graph representation: Simulation processes the earliest event, advances simulated time, and enqueues successors when their dependencies are resolved.Deterministic nodes schedule fixed completion events, while nondeterministic nodes can have completion events replaced after resource-state changes.
  • Execution graph representation: Logical nodes can grow the execution graph at runtime as serving decisions become known, accommodating dynamic request paths and dependencies.This addresses runtime-dependent batch composition, placement, and request stages.

C. Modular Control Layer

Simthesizer partitions serving control into modular components that own distinct roles and can be composed through structured configuration, while the framework also supports extensible timing and serving implementations.

  • Componentized control: Control components separately own roles such as request routing, batch scheduling, and model execution, including their policy state and logical nodes.This avoids placing every serving decision in one shared handler.
  • Configuration and composition: Structured configuration assembles components into a complete simulator by selecting implementations, nesting subcomponents, and setting implementation-specific parameters.Figure 5 illustrates interfaces, concrete implementations with encapsulated state, and configuration composition.
  • Configuration and composition: Swapping a scheduler implementation does not require changes to the system component or execution layer.SingleInstance holds its scheduler through the interface, while ChunkedPrefill privately tracks its prefix-caching flag.
  • Built-in components: The control layer includes interfaces for request routing, orchestration, batch scheduling, model execution, model description, and compute and network timing.Built-in implementations cover single- and multi-instance serving, disaggregation, MoE serving, and common routing policies.
  • Agent integration: Synthesizer agent uses the modular interfaces to extend the simulator while harnesses expose constraints and organize specification, implementation-map, and validation artifacts.The workflow shifts human attention toward performance-critical ambiguities and review.

A. Simulator-Specific Synthesis Requirements

Synthesizer agent structures feature requests into explicit, traceable, and fidelity-appropriate simulator decisions before implementation, using staged documents and guardrails to manage ambiguity.

  • Requirements: Initial feature requests often leave performance-critical modeling decisions implicit, so synthesis must resolve them into simulator-facing semantics.These decisions affect simulation time, contention, state lifetime, dependencies, and metric boundaries.
  • Requirements: Semantic completeness requires explicit quantities and assumptions governing simulation time, resource contention, state lifetime, dependency structure, and metric boundaries.Unresolved performance-relevant choices must be surfaced for human review.
  • Requirements: Structural alignment maps each modeling decision to the abstraction responsible for its state transition, graph update, or resource interaction.This preserves a traceable connection between specification and generated implementation.
  • Requirements: Modeling fidelity requires matching implementation detail to the target performance question and documenting assumptions, evidence, and validity boundaries for approximations.Permitted approximations include aggregate multipliers and statistical proxies.
  • Lowering process: The lowering process uses task-design, sim-mapping, and sim-dev to produce a specification, map decisions to simulator abstractions, and implement the changes.Guardrails detect undeclared approximations, unexplained constants, and deviations from the implementation map.
  • Lowering process: A structured task-design summary records target configuration and modeling inputs such as chunked prefill, prefix caching, speculative width K = 2, and stochastic acceptance sampling.The supplied examples identify EAGLE-3 as the algorithm reference.

C. Evidence-Guided Validation

Simthesizer validates generated simulator behavior through real-system traces when available and technical references otherwise, with speculative decoding serving as a concrete extension case.

  • Validation regimes: Evidence-guided validation addresses the gap between consistency with a specification and faithful modeling of the target mechanism.It uses two complementary validation regimes.
  • Validation regimes: Trace-guided validation compares a candidate simulator with the real system when real-system measurements are available.The supplied passage introduces this regime but does not provide its detailed comparison outcome.
  • Case study: The implementation map for speculative decoding is presented as Table II.The supplied table passage identifies the table but does not expose its contents.
  • Validation regimes: Reference-guided validation grounds modeling decisions in technical evidence when comparable measurements are unavailable, including for unreleased hardware or novel mechanisms.Criteria can be derived from reference implementations.
  • Case study: EAGLE-style speculative decoding is used as a case study in extending the simulator with a drafter model, target-model verification, and accepted-token progress.The mechanism generates draft tokens rapidly and verifies them in one target-model inference iteration.

A. Resolving Simulator Semantics

Synthesizer agent resolves underspecified feature requests into explicit semantics, maps them onto reusable simulator components, and validates executable extensions against reference behavior. In one speculative-decoding case, trace-guided validation exposed a semantic error and enabled a localized correction.

  • Semantic resolution: Synthesizer agent derives explicit semantic variables from underspecified algorithmic requests, including verification width, remaining output length, and accepted draft length.Task-design resolves unspecified modeling decisions into simulator semantics.
  • Component mapping: The sim-mapping stage records how simulator-facing specifications map onto existing Simthesizer components and interfaces.Mechanism-specific logic is localized within reusable components rather than added as separate machinery.
  • Implementation: The sim-dev stage translates the implementation map into localized code changes without introducing new events, node types, graph APIs, or execution loops.Most simulator components remain unchanged.
  • Validation: 13.4% throughput overestimation was traced to incorrect verification-width semantics that underestimated required compute and communication.The validation evidence isolated the issue from component mapping and DAG execution.
  • Validation: A corrected candidate reduced throughput divergence to 6.7% and aggregate-throughput underestimation to 4.7% on the same validation trace.The validation report fed the semantic correction back into another synthesis iteration.

D. Joint Effect

Simthesizer combines reusable simulation components with an agent-driven extension workflow and evaluates that workflow against baseline simulators under controlled conditions. The evaluation measures fidelity, harness effects, simulation speed, and coding-agent generality.

  • Joint design: Simthesizer combines reusable scheduling, compute, cache-management, and DAG-execution components with agent mapping of underspecified behavior.In speculative decoding, mechanism-specific logic is localized to the acceptance policy and scheduler while shared execution and resource-modeling paths are preserved.
  • Evaluation scope: The evaluation asks whether Simthesizer models complex workloads, supports unsupported features, benefits from its harness, runs quickly, and generalizes across coding agents.These five questions define the evaluation scope.
  • Experimental comparison: Figures 9–11 compare throughput, error rates, and simulation time for the three simulators after each is independently extended with the same feature.The shared convention ensures each panel compares distinct simulators receiving the same extension task.
  • Experimental comparison: The study compares Simthesizer with LLMServingSim2.0 and Vidur using three unsupported feature-extension tasks under identical prompts, references, specifications, profiling data, coding agent, and harnesses.The tasks are FP8 KV cache quantization, EAGLE3 speculative decoding, and hybrid Mamba support, evaluated over repeated independent trials.

B. Evaluation Results

Simthesizer follows real-system behavior on complex workloads and retains higher extension fidelity than baseline simulators, especially for mechanisms requiring coordinated changes. Its harness improves extension accuracy, while its simulations are substantially faster and its results generalize across coding agents.

  • Complex-workload fidelity: Simthesizer closely matches vLLM for dense agentic workloads, while MoE discrepancies are more visible because routing and synchronization are statistically modeled.It nevertheless captures interactions between generation and tool calls and follows real-system performance trends.
  • Complex-workload fidelity: TTFT and ITL errors exceed TPOT errors because network traffic and kernel-launch delays are difficult to simulate precisely.Lower TPOT error indicates accurate modeling of batching and scheduling, while overall behavior remains closely reproduced.
  • Extension fidelity: Simthesizer achieves highest accuracy on speculative decoding and hybrid Mamba support, whose coordinated scheduling, model-structure, and performance changes challenge fixed pipelines.KV cache quantization yields low error across all simulators because it mainly requires memory tracking and quantization-overhead modeling.
  • Extension fidelity: 2.51% average throughput error versus 6.03% for baseline-built extensions shows higher extension fidelity when the coding agent, harness, specifications, and profiling data are identical.The controlled comparison isolates the contribution of the underlying simulator.
  • Harness impact: Without the Synthesizer agent harness, simulation error rises by 1.65×–3.69× across throughput and latency metrics.The harness supplies knowledge, resolves required specifications before implementation, and enforces self-validation.
  • Coding-agent generality: Codex- and Claude Code-produced speculative-decoding simulators achieve average errors of 4.1% and 4.9%, respectively, across throughput, TTFT, TPOT, and ITL.Both follow the vLLM real-system trajectory under the same harness design.

VIII. RELATED WORK

Prior simulators rely on human developers to encode system behavior into fixed pipelines, whereas Simthesizer uses a unified dynamic DAG and agent-driven lowering to accommodate evolving serving mechanisms. The paper positions this pairing as a scalable approach beyond LLM serving.

  • Existing simulators: Existing LLM and system simulators generally encode system behavior in advance through human-developed simulation pipelines.This pattern spans hardware-level LLM simulators, system-level serving simulators, and distributed-training simulators.
  • Simthesizer: Simthesizer represents the complete serving workflow as uniform elements of a unified dynamic DAG and adds mechanisms by connecting nodes through agent-driven lowering.This contrasts with repeated manual restructuring and revalidation of fixed pipelines.
  • Broader implication: Simthesizer combines composable workflow infrastructure with a harnessed coding agent that lowers high-level specifications into executable extensions.The framework targets fast-evolving, non-monolithic LLM serving.
  • Broader implication: The reported results suggest that this infrastructure-and-lowering pairing offers a scalable path for simulator development in other rapidly evolving systems domains.This is presented as a supported implication of the paper’s conclusion.
Loading 2608.24650v1…