PaperlayerView exact revision on arXiv

Source-linked AI summary

ServeTwin: A Benchmark-Validated Simulator for Distributed LLM Architecture Exploration

Sungjoon Park, Changue Jung, Kyungno Joo, Mincheol Kang, Jaehyung Ahn, Sehwan Lee, Sangjoon Kim

arXiv:2610.02732v1cs.PFcs.DC

TL;DR

Distributed LLM serving design is costly to evaluate, and existing simulators cover only subsets of the capabilities needed for dynamic serving exploration. ServeTwin couples a stateful serving loop with specification-driven timing; it reproduces benchmark behavior and supports predictions from hardware specifications.

  • Problem

    Physical-cluster evaluation is costly, while existing simulators provide only subsets of stateful closed-loop execution, profiling-free timing prediction, and unmodified benchmark compatibility.

    Source: §1 · p. 1

  • Method

    ServeTwin couples a stateful serving loop that tracks scheduling and KV-cache lifecycles with iSTAGE, which derives analytical execution traces from hardware and model specifications.

    Source: §1 · p. 2

  • Results

    ServeTwin achieves 3.6% mean error on InferenceX frontiers and reproduces queue and cache-reuse dynamics in multi-turn scenarios; its GB200 NVL72 prediction is within a 2% margin.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    Hardware upgrade priorities must align with workload states and service targets, and ServeTwin can generate serving curves from pre-silicon specifications.

    Source: §6 · p. 12

    GB200 results are transferred platform-model predictions rather than empirical measurements.

    Source: §4.8 · p. 11

Abstract

from arXiv · show

Evaluating distributed LLM serving designs on physical clusters is costly. Yet existing simulators provide only subsets of the capabilities needed for realistic design exploration: stateful closed-loop execution, timing prediction without profiling target hardware, and direct execution of unmodified serving benchmarks. We present ServeTwin, a closed-loop simulator that couples specification-driven analytical timing with a stateful serving loop. This coupling captures feedback among request completion, scheduling, queue state, and KV-cache evolution, including prefill-decode disaggregation and multi-turn agent workloads. ServeTwin avoids target-hardware operator profiling through iSTAGE, an analytical trace generator that derives per-iteration traces from model and engine specifications while representing ragged batches and discrete engine mechanisms such as CUDA-graph batch padding. It further decomposes execution time into component-owned throughput, scheduler, and runtime costs. This ownership lets unaffected parameters transfer across platforms and confines recalibration to changed hardware or software components. ServeTwin implements a vLLM-compatible interface and runs unmodified serving benchmarks. Against real deployments, it reproduces InferenceX's steady-state throughput-interactivity frontier with a 3.6% mean error and predicts LMBenchmark's multi-turn performance with a 9.9% error while tracking KV-cache evolution. Pre-silicon sweeps reveal that the preferred HBM bandwidth-capacity tradeoff reverses across workload states and that increasing concurrency shifts the bottleneck from memory bandwidth to scheduler and runtime overheads. Together, these capabilities enable practical exploration of distributed LLM serving systems before target hardware is available.

1 Introduction

Distributed LLM serving needs simulators that capture feedback among scheduling, queues, and KV-cache state, including disaggregation and multi-turn workloads. ServeTwin combines a stateful serving loop with specification-driven timing and validates its predictions against serving benchmarks.

1 Introduction

  • ServeTwin uniquely combines stateful closed-loop execution, profiling-free timing prediction, and compatibility with unmodified serving benchmarks.The comparison categorizes prior tools as full, partial, or absent for each capability.Source: Table 1 · p. 1
  • Existing simulators either model static workloads without queueing feedback or reproduce scheduling using rigid hardware profiles, and some fail on disaggregated multi-turn workloads.LLMServingSim lacks a staged-KV waiting state, causing multi-turn requests to crash when decode capacity is unavailable.Source: §1 · pp. 1–2
  • ServeTwin couples a stateful serving loop with specification-derived timing, feeding execution time back into scheduling and completing multi-turn benchmarks under deployment-equivalent KV budgets.Its loop enforces admission control, cross-worker KV transfers, and multi-turn state.Source: §1 · p. 2
  • iSTAGE addresses ragged batches, discrete runtime mechanisms, and the need for lightweight trace generation through statistics-preserving binding, first-order mechanism modeling, and trace reuse.Naïve padding to maximum dimensions can overestimate execution time by up to 146×.Source: §1 · p. 2
  • 3.6% mean error on InferenceX frontiers, while ServeTwin captures multi-turn transients within a 10% error margin.InferenceX evaluates steady-state throughput and interactivity; LMBenchmark adds multi-turn dynamics.Source: §1 · p. 2
  • Component-owned cost decomposition separates specification-driven throughput from host-control and runtime costs, allowing recalibration to focus on the component that changes.The paper also validates across benchmarks and explores workload-dependent HBM trade-offs and shifts toward runtime-control bottlenecks.Source: §1 · pp. 2–3

2 Background and Motivation

LLM serving is stateful: memory availability and runtime mechanisms shape each batch, while disaggregated prefill and decode workers depend on KV transfer. The section describes the benchmarks and simulator gaps that motivate ServeTwin’s focus on occupancy-coupled state.

2.1 LLM Serving as a Stateful System

  • Serving is state-dependent: KV memory determines whether requests are admitted, queued, or preempted, and runtime mechanisms can reshape physical execution.Continuous batching and chunked prefill reconstruct batches each iteration; prefix caching reuses history.Source: §2.1 · p. 3

2.2 Prefill/Decode Disaggregation and KV Transfer

  • Disaggregated prefill and decode require KV transfer; when decode memory is full, KV remains staged on the prefill worker and creates upstream backpressure.Production pull mechanisms fetch KV blocks only when decode workers have sufficient free capacity.Source: §2.2 · p. 3

2.3 Benchmarks Used in This Paper

  • InferenceX evaluates steady-state throughput-interactivity balance, while LMBenchmark captures multi-turn admission pressure and transient delays against physical deployment telemetry.InferenceX comparison uses public hardware performance data; LMBenchmark validation uses an in-house DGX H100 with Dynamo and vLLM.Source: §2.3 · p. 3

2.4 Existing Simulation Approaches

  • Serving simulators rely on hardware-profiled latencies and omit key stateful KV semantics, while analytical models use specifications but evaluate static workloads offline.STAGE models prescribed workload graphs rather than scheduler-generated ragged batches and a stateful serving loop.Source: §2.4 · p. 3

2.5 Why Existing Serving Simulation Is Insufficient

  • At the deployment-equivalent KV budget, LLMServingSim aborts because it cannot retain completed-prefill KV on the producer while waiting for decode capacity; this is a semantic, not calibration, failure.ServeTwin completes the unmodified multi-turn benchmark under the same budget, with all methods using the same calibrated compute profile.Source: §2.5 · p. 3

3 ServeTwin Design

ServeTwin combines a stateful serving loop with specification-driven execution modeling to simulate dynamic request, scheduling, and KV-cache behavior. Its cost decomposition assigns delays to system components, supporting selective recalibration when those components change.

3.1 Overview

  • The closed loop batches requests, generates execution traces, simulates their timing, then advances the clock and updates state for the next iteration.Unmodified clients connect through a wall-clock-to-simulated-time interface.Source: §3.1 · pp. 3–4

3.2 Serving-State Simulation

  • ServeTwin models continuous batching, chunked prefill, prefix caching, and capacity-driven preemption with explicit KV lifecycles and multi-turn session state.Its registry includes occupancy-coupled admission, tiered memory, and a dedicated session layer.Source: §3.2 · p. 4

3.3 Disaggregated KV Lifecycle

  • Decode-side memory saturation blocks admission, while consumer-driven KV pulls gate decoding; the resulting stalls can build queues and network backlogs.The model supports full or layer-wise transfers and suffix recomputation.Source: §3.3 · p. 4, Figure 3 · p. 4

3.4 Multi-turn Sessions as Simulator State

  • Sessions issue each next turn only after the prior response completes, while accumulating context enables prefix-cache hits and minimal delta transfers.Tiered KV memory and bandwidth-constrained host-device movement are also modeled.Source: §3.4 · p. 4

3.5 Spec-Driven Execution Modeling with iSTAGE

  • iSTAGE derives profile-free timing from hardware specifications and binds ragged batches to aggregate-preserving shapes, yielding 3.7% median error versus 268% for max-folding on mixed batches.Across all iterations, including pure decode, median error is 1.1%.Source: §3.5 · p. 5
  • iSTAGE models discrete CUDA-graph capture padding as real work and reuses cached shapes, folded layers, and repeated decode traces to keep generation lightweight.Layer folding remains exact by re-adding per-collective network latencies.Source: Figure 4 · p. 4, §3.5 · p. 5

3.6 Component-Driven Latency Modeling

  • Component ownership decomposes iteration time into throughput, scheduler, and runtime costs, mapping each delay to an operation and its scaling variables.The categories include GPU compute, memory and network transfer; CPU control and padding; and KV traversal, kernel launches, and communication setup.Source: §3.6 · pp. 5–6
  • Freezing parameters unless their owning component changes supports cross-platform transfer and limits recalibration to the affected component.The model structurally isolates costs to avoid arbitrary curve fitting.Source: §3.6 · p. 6, Table 3 · p. 6

4 Evaluation

The evaluation tests attribution, transferability, fidelity, simulation cost, and design exploration across specified model, hardware, and serving setups. Calibration uses unmodified benchmarks and reuses component-specific parameters unless their owning components change.

4.1 Experimental Setup

  • The evaluation frames five questions on cost attribution, parameter transfer, serving fidelity, simulator cost, and whether simulation changes pre-silicon design decisions.Primary configurations use DeepSeek-R1 on H200 and disaggregated H200/B200 deployments; transient validation uses Qwen3-32B on 2×H100.Source: §4.1 · p. 6
  • The setup isolates KV-transfer traffic and analyzes 44,220 iterations across 10 runs, including decode, chunked prefill, and multi-turn contexts up to 21K tokens.One GPU serves prefill and one decode, restricting NVLink traffic to KV transfers.Source: §4.1 · p. 7

4.2 Calibration Methodology

  • ServeTwin validates unmodified endpoint benchmarks by calibrating on a subset of public InferenceX measurement points and evaluating held-out points, while reusing component-specific overhead parameters unchanged unless their owning components change.Disaggregation adds structural terms, including prefill-to-decode pacing and DPA residuals. On a different software snapshot, remeasured host control for disaggregated H200 was within 0.61 ms of the B200 baseline.Source: §4.1 · p. 7, §4.2 · p. 7

4.3 Sensitivity Analysis of Cost Terms

Ablations show that distinct cost families matter under their intended serving conditions, while selective transfer predicts changed deployments by updating only affected parameters. The reported transfer results include low average errors and disclose larger errors at extreme concurrency.

4.3 Sensitivity Analysis of Cost Terms

  • Removing GEMM utilization raises prefill error by 56 pp, removing per-step host control raises small-batch decode error by 47 pp, and removing DPA synchronization raises DPA-decode error by 40 pp.Other terms, including KV traversal and collective costs, also dominate errors under their corresponding conditions.Source: §4.3 · p. 7

4.4 Selective Transfer

  • Across hardware, stack, worker-count, and platform changes, selective parameter updates predict held-out data, with 3.1% average error for B200-to-disaggregated-H200 transfer and anchors matched within 2% on GB200 NVL72.The P→D pacing parameter transfers unchanged when scaling from one to two prefill workers within the same handoff-path class.Source: §4.4 · p. 8
  • Extreme-concurrency transfer errors remain 7–12% where batching amortizes per-request overheads sublinearly, at approximately n^0.6.The authors report these errors transparently rather than using arbitrary overfitting.Source: §4.4 · p. 8

4.5 End-to-End Steady-State Fidelity: InferenceX

ServeTwin reproduces InferenceX steady-state behavior across H200, B200, and GB200 environments with low reported error. Held-out accuracy closely tracks calibration accuracy, and dense-model compute efficiency follows the reported arithmetic-intensity comparison.

4.5 End-to-End Steady-State Fidelity: InferenceX

  • Figure 6 validates the simulator across H200, B200, and GB200 environments: across 27 operating points, mean error was 3.6%, with every prediction within a 10% margin.Source: §4.5 · p. 8
  • 3.7% held-out validation error closely matches 3.5% calibration error, indicating similar prediction error on those reported point sets.Source: §4.5 · p. 8
  • Qwen3-32B yields 3× higher compute efficiency than the MoE baseline, matching their arithmetic-intensity ratio.The passage attributes the efficiency pattern to architectural properties rather than tuning variables.Source: §4.5 · p. 8

4.6 Transient Serving Fidelity: LMBenchmark

ServeTwin is evaluated on transient LMBenchmark scenarios with changing cache reuse and queue pressure. LLMServingSim matches baseline TTFT but diverges substantially under pressure scenarios and across ITL measurements.

4.6 Transient Serving Fidelity: LMBenchmark

  • The evaluation uses Qwen3-32B FP8 on two H100 GPUs with one prefill and one decode worker, vLLM, prefix caching, and baseline plus three workload variants.The variants double input sequence length, multiply output length by ten, or increase users to 20; users append to and resend accumulated conversations.Source: §4.6 · p. 8, Figure 7 · p. 9
  • ISL×2 peaks the prefill queue at 16 requests as larger histories increase KV occupancy and cold prefills; USERS×20 lowers cache-hit rates through competing per-user histories.In USERS×20, histories do not share prefixes, so KV-pool evictions occur before reuse.Source: §4.6 · p. 8
  • OSL×10 prolongs decoding, causing warm-up prefill accumulation and a decode-admission spike; cache-hit rates rise more slowly than baseline.Higher eventual cache-hit rates reduce prefill work but do not eliminate the extended decode phase.Source: §4.6 · pp. 8–9
  • LLMServingSim matches baseline TTFT within 10%, but on ISL×2 and USERS×20 its TTFTs are 96–99.7% below measurement; ITL is overestimated by 79–137% in all scenarios.For OSL×10, replay underestimates TTFT by 42% and chained adaptation overestimates it by 38%.Source: §4.6 · p. 9

4.7 Simulation Efficiency

Trace reuse and analytical shortcuts make ServeTwin’s experiments practical on a commodity CPU.

4.7 Simulation Efficiency

  • Trace-generation overhead is reduced by reusing repeated trace expressions, jumping across repeated Transformer layers, and caching shape buckets.Layer-wise repetition provides a further 4.9× speedup, and experiments including calibration sweeps run deterministically on a single commodity CPU in minutes.Source: §4.7 · p. 9
  • Layer-wise repetition lets ServeTwin analytically jump to the end of all layer executions instead of simulating individual layers.This optimization provides a further 4.9× reduction in trace-generation overhead.Source: §4.7 · p. 9
  • A shape cache groups slightly varying tensor shapes into size-based buckets that share entries rather than caching every shape separately.Repeated end-to-end trace expressions are also generated once and reused.Source: §4.7 · p. 9

4.8 Pre-Silicon Design Exploration

Pre-silicon sweeps show that bandwidth, capacity, and software overheads matter differently across workload states and concurrency levels. Decode-step decomposition further shows that hardware gains depend on batch size.

Pre-Silicon Design Exploration

  • The design explorations are simulated sensitivity analyses, not equal-cost hardware comparisons or measurements of unreleased silicon.The GB200 decode results are transferred platform-model predictions; its per-step Scheduler cost is reduced from 1.8 ms to 0.635 ms.Source: §4.8 · pp. 10–11

Bandwidth vs. Capacity

  • For multi-turn W2, 1.4× memory pool capacity improves throughput by 22% and reduces median TTFT from 12.55 s to 0.068 s, while 1.4× bandwidth has little throughput effect.Cache hits rise from 22% to 93%, and median queueing delay falls from 10.46 s to approximately 0.01 s.Source: Figure 8 · p. 10, §4.8 · p. 10
  • For W1, increasing bandwidth to 1.4× baseline improves throughput by 24%, while increasing the memory pool provides no throughput benefit; reducing it to 0.7× baseline lowers throughput by 5% under this fixed workload setting.Source: §4.8 · p. 10
  • Across W1 and W2, the benefits of bandwidth and capacity depend on active KV demand, prefix reuse, and the stages limiting serving performance.The sensitivity analysis varies bandwidth and usable memory pool separately on Qwen3-32B FP8 in a two-GPU H100 1P1D configuration.Source: §4.8 · p. 10

Bandwidth vs. Scheduler and Runtime Costs

  • At concurrency 2048, tripled bandwidth makes raw token generation 60% faster but stalls overall throughput and degrades median TTFT to 52.96 s; cutting selected Scheduler and Runtime costs by 40% raises throughput 60% and halves TTFT to 22.92 s.At concurrency 128, bandwidth increases throughput by 28–63%; the study uses aggregated and disaggregated H200 configurations.Source: §4.8 · pp. 10–11, Figure 9 · p. 11

Batch-Dependent Limits to Hardware Gains

  • At B=1, Runtime Cost accounts for 63–77% of step time; H200-to-B200 Throughput Cost falls from 1.67 ms to 1.09 ms, but total step time only falls from 9.34 ms to 8.76 ms.The comparison uses aggregated TP8 DeepSeek-R1 FP8 with a 1536-token FP8 KV cache.Source: Figure 10 · p. 11, §4.8 · p. 11
  • At B=32, H200-to-B200 Throughput Cost falls from 10.94 ms to 6.64 ms, reducing total step time from 21.34 ms to 17.04 ms.The decomposition separates analytical trace execution, exposed host work, and runtime costs.Source: Figure 10 · p. 11, §4.8 · p. 11

5 Related Work

ServeTwin is distinguished from prior simulators by its modeling of mechanism-level costs and state-dependent queueing behavior, alongside direct modeling of disaggregated serving mechanisms.

Simulators and Analytical Models

  • Unlike LIMINAL [5], which bounds decode performance using a single lumped overhead term, ServeTwin isolates specific mechanism costs to accurately model state-dependent queueing behaviors.Most simulators, including APEX [7], are validated solely on custom internal testbeds.Source: §5 · p. 11

Overhead Characterization

  • ServeTwin complements KernelSight-LM’s [18] direct instrumentation of host overheads by analytically inferring overheads from public measurements and verifying component-level parameter transfer.Source: §5 · pp. 11–12

Disaggregated Systems

  • ServeTwin models mechanisms underlying disaggregated serving architectures pioneered by Splitwise, DistServe, and Mooncake.Source: §5 · p. 12

6 Conclusion

ServeTwin combines a KV-lifecycle state machine with specification-driven timing, achieving benchmark validation and enabling pre-silicon serving-curve exploration.

6 Conclusion

  • 3.6% mean error on InferenceX frontiers validates ServeTwin against independent public benchmarks.It also reproduces queue and cache-reuse dynamics in multi-turn scenarios.Source: §6 · p. 12
  • Within a 2% margin, ServeTwin predicts GB200 NVL72 performance by updating calibration parameters according to architectural changes.This transfers the simulator to new platforms.Source: §6 · p. 12
  • ServeTwin enables architects to generate serving curves from pre-silicon specifications, with hardware upgrade priorities aligned to workload states and service targets.Source: §6 · p. 12
Loading 2610.02732v1…