PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure

Jaehong Cho, Hyunmin Choi, Guseul Heo, Jongse Park

arXiv:2602.23036v2cs.DCcs.AI

TL;DR

Existing simulators do not jointly capture runtime interactions across heterogeneous hardware and disaggregated LLM serving. LLMServingSim 2.0 unifies these dynamics in a system-level simulation loop and reproduces key metrics with 0.95% average error.

  • Problem

    Existing simulators incompletely capture runtime interactions among heterogeneous hardware, disaggregated architectures, scheduling, memory behavior, and interconnect effects in LLM serving.

  • Method

    LLMServingSim 2.0 uses a unified runtime-driven loop with profile-based hardware modeling to jointly simulate serving decisions, heterogeneous devices, memory, and power.

  • Results

    0.95% average error across key performance, memory, and power metrics validates LLMServingSim 2.0 against real deployments.

  • Takeaways & Limitations

    LLMServingSim 2.0 provides a practical framework for analyzing hardware–software interactions and exploring heterogeneous, disaggregated LLM serving designs.

  • Takeaways & Limitations

    Existing simulators remain limited in capturing memory management and heterogeneous deployments, motivating the broader modeling scope of LLMServingSim 2.0.

Abstract

from arXiv · show

Large language model (LLM) serving infrastructures are undergoing a shift toward heterogeneity and disaggregation. Modern deployments increasingly integrate diverse accelerators and near-memory processing technologies, introducing significant hardware heterogeneity, while system software increasingly separates computation, memory, and model components across distributed resources to improve scalability and efficiency. As a result, LLM serving performance is no longer determined by hardware or software choices in isolation, but by their runtime interaction through scheduling, data movement, and interconnect behavior. However, understanding these interactions remains challenging, as existing simulators lack the ability to jointly model heterogeneous hardware and disaggregated serving techniques within a unified, runtime-driven framework. This paper presents LLMServingSim 2.0, a unified system-level simulator designed to make runtime-driven hardware-software interactions in heterogeneous and disaggregated LLM serving infrastructures explicit and analyzable. LLMServingSim 2.0 embeds serving decisions and hardware behavior into a single runtime loop, enabling interaction-aware modeling of batching, routing, offloading, memory, and power. The simulator supports extensible integration of emerging accelerators and memory systems through profile-based modeling, while capturing dynamic serving behavior and system-level effects. We validate LLMServingSim 2.0 against real deployments, showing that it reproduces key performance, memory, and power metrics with an average error of 0.95%, while maintaining simulation times of around 10 minutes even for complex configurations. These results demonstrate that LLMServingSim 2.0 provides a practical bridge between hardware innovation and serving-system design, enabling systematic exploration and co-design for next-generation LLM serving infrastructures.

I. INTRODUCTION

LLMServingSim 2.0 is a unified, runtime-driven simulator for analyzing interactions among heterogeneous hardware, disaggregated serving structures, and serving decisions. It combines interaction-aware modeling, extensible hardware profiles, and power-aware evaluation to support system-level exploration of emerging LLM serving infrastructures.

  • Motivation: Heterogeneous devices and disaggregated resources interact at runtime through scheduling, memory behavior, and interconnect contention, making isolated hardware or software analysis insufficient.Modern deployments combine diverse accelerators and separate computation, memory, and model components across distributed resources.
  • Core framework: LLMServingSim 2.0 embeds serving decisions and hardware behavior in one runtime-driven loop, allowing batching, routing, placement, and offloading to adapt dynamically to system conditions.The simulator models how these decisions evolve and propagate over time.
  • Core framework: The simulator jointly represents heterogeneous accelerators, multi-tier memory, and disaggregated architectures so their combined system behaviors can be evaluated coherently.This unified representation avoids isolating hardware and software effects when studying disaggregated serving techniques.
  • Runtime dynamics: Runtime-driven modeling exposes temporal effects such as queueing, contention amplification, and phase-dependent behavior arising from dynamic request flows and resource contention.Performance emerges from runtime conditions rather than static or a priori configurations.
  • Extensibility: Profile-based operator modeling enables low-effort integration of emerging accelerators and memory technologies without restructuring the serving model.This supports rapid exploration of new hardware–software combinations.
  • Evaluation: Power modeling links compute, memory, and data-movement activities to power characteristics, enabling joint analysis of performance, energy efficiency, heterogeneity, and disaggregation choices.The paper evaluates the simulator’s accuracy, efficiency, and practical usefulness against real deployments using serving metrics including throughput, TTFT, and TPOT.

II. BACKGROUND … III. MOTIVATION

LLM serving combines compute-intensive prefill with memory-intensive decode across dynamic, parallelized deployments. Heterogeneous accelerators and multi-tier memory make execution, data movement, and device behavior tightly interdependent, motivating unified modeling tools.

  • A. LLM Inference Serving: Workflow and System Structure: LLM inference uses compute-intensive prefill and memory-intensive decode, with decode constrained primarily by memory bandwidth, capacity, and KV-cache locality.Prefill processes input sequences through dense matrix multiplications, whereas decode generates tokens autoregressively through repeated KV-cache accesses.
  • A. LLM Inference Serving: Workflow and System Structure: Data-center serving concurrently runs tens to hundreds of model instances, while dynamic batching and queuing create time-varying compute and memory utilization.Serving frameworks use tensor, pipeline, and data parallelism, along with expert parallelism for mixture-of-experts models.
  • B. Heterogeneous Accelerators and Memory Hierarchies: Modern LLM clusters combine different GPU generations, NPUs, TPUs, Processing-in-Memory accelerators, and Compute Express Link devices.These platforms differ in compute capability, on-device memory bandwidth, memory capacity, and interconnect throughput.
  • B. Heterogeneous Accelerators and Memory Hierarchies: Memory hierarchy plays a central role in LLM inference performance.The supplied passage identifies memory hierarchy as central, while the adjacent discussion describes its multiple tiers and KV-cache effects.
  • B. Heterogeneous Accelerators and Memory Hierarchies: KV-cache accesses span accelerator HBM, host DRAM, storage devices, and large CXL memory pools with distinct latency and bandwidth characteristics.KV-cache placement, reuse, and migration directly influence decode latency and throughput; MoE expert weights may also be partitioned or offloaded.
  • III. MOTIVATION: The diverse compute-memory stack creates complex interactions among execution behavior, data movement, and device heterogeneity, motivating modeling tools that capture modern serving characteristics.These interactions span heterogeneous accelerators and multi-tier memory systems.

A. Modeling Complex Runtime Dynamics in LLM Serving … C. Model Serving Group

LLMServingSim 2.0 models LLM serving as a runtime-driven interaction among dynamic requests, heterogeneous hardware, memory, routing, and scheduling. Its unified architecture combines configurable model-serving groups with system-level simulation to support hardware-software co-design and architectural exploration.

  • A. Modeling Complex Runtime Dynamics in LLM Serving: Runtime dynamics couple request arrivals, queueing, fluctuating batches, adaptive placement, KV-cache movement, prefix-cache behavior, and MoE routing.These interactions determine serving metrics such as time-to-first-token, as indicated by the supplied passage.
  • B. Limitations of Existing LLM Serving Simulators: Hardware-centric simulators model accelerator execution and communication but omit dynamic serving behaviors such as request arrivals, batching evolution, and KV-cache reuse.The passage identifies LLMCompass and ADOR as examples of hardware-centric simulators with static-execution emphasis.
  • C. Need for a Unified Simulator: A unified simulator is required because heterogeneous accelerators, multi-tier memory, parallelism, prefix caching, routing, and disaggregation interact across system layers.The stated motivation is to enable hardware-software co-design, policy evaluation, and architectural exploration.
  • A. Overview: LLMServingSim 2.0 accepts workload, cluster, and hardware-performance specifications describing request patterns, deployment resources, serving policies, and device behavior.The workload includes arrival rates and execution traces; the cluster includes node, CPU, memory, placement, and serving-policy configurations.
  • A. Overview: The runtime loop routes requests to model-serving groups, generates batch execution graphs, and evaluates them with the System Simulator.The architecture connects the Request Router, MSGs, and System Simulator iteratively during execution.
  • C. Model Serving Group: Each Model Serving Group serves one LLM instance through a configurable device pool that can combine GPUs, NPUs, CXL-attached devices, and PIM-equipped memory channels.The device pool provides the execution substrate for operator mapping and runtime serving decisions.
  • C. Model Serving Group: Within each MSG, request queues, batch scheduling, operation mapping, and operation scheduling coordinate execution under memory, KV-cache, parallelism, and offloading constraints.The operation mapper can place attention on PIM or offload experts and KV caches to selected memory devices, while the scheduler constructs dependency-aware execution DAGs.
  • C. Model Serving Group: The System Simulator extends ASTRA-sim and Chakra for heterogeneous compute fabrics, operator-driven inference graphs, dynamic memory behavior, and PIM-accelerated execution.The refined memory model represents bandwidth contention, KV movement, memory sharing, device-to-CXL memory tiers, and additional PIM operations.

V. SERVING TECHNIQUES AND USE-CASE MODELING · A. Model Serving Group-Level Techniques · B. System-Level Serving Techniques

LLMServingSim 2.0 models heterogeneous, disaggregated serving through runtime-generated execution graphs that capture operator placement, data movement, synchronization, and communication. It supports operator-granular offloading, MoE expert placement, prefill-decode disaggregation, and multi-tier prefix caching across configurable resources.

  • A. Model Serving Group-Level Techniques: A single model serving group can combine heterogeneous devices and place operators on specialized devices according to serving policies.At runtime, the operation mapper assigns each operator to an appropriate device at operator granularity.
  • A. Model Serving Group-Level Techniques: Attention offloading maps attention operators to PIM while executing remaining operators on GPUs when both device types are available.This configuration is illustrated for a device pool containing GPUs and PIM modules.
  • A. Model Serving Group-Level Techniques: Required activation and KV-cache transfers, synchronization, and cross-device operations are automatically inserted into execution graphs and included in latency modeling.For attention offloading, activations and KV caches move from GPUs to PIM before execution and results return afterward.
  • A. Model Serving Group-Level Techniques: MoE tokens are routed to experts that may reside on different devices or be evicted to host memory, with loading and communication translated into execution graphs.The System Simulator evaluates the resulting graph and accounts for latency from expert loading and cross-device communication.
  • B. System-Level Serving Techniques: Prefill and decode serving groups can use different hardware, memory hierarchies, and interconnects while serving the same model.The Execution Planner instantiates multiple groups, and policies support arbitrary M:N mappings between prefill and decode groups.
  • B. System-Level Serving Techniques: The request router dispatches requests to prefill groups, whose schedulers insert layer-wise KV-cache transfers to decode groups and model network-dependent latency.The memory model handles KV movements, while the System Simulator evaluates inter-node communication according to the configured network topology.
  • B. System-Level Serving Techniques: Multi-tier radix-tree prefix caches can be instantiated across device and shared host-memory tiers, enabling systematic exploration of caching policies.Prefix hits reduce effective execution latency, while missing KV caches trigger automatically inserted layer-wise transfers from the appropriate tier.

VI. METHODOLOGY · VII. EVALUATION · A. Validation with Real Serving System

LLMServingSim 2.0 is evaluated across heterogeneous GPU and TPU platforms, models, and serving configurations using profiled hardware backends and simulator baselines. Real-system validation shows close agreement in throughput, power, memory usage, and cache behavior.

  • VI. METHODOLOGY: Three platforms—RTX A6000, H100-SXM-80GB, and TPU-v6e-1—evaluate dense and MoE models with vLLM as the serving framework.The study covers Llama 3.1-8B, Phi-mini MoE, and Llama 3.1-70B across the listed systems.
  • VI. METHODOLOGY: Operator-level profiling integrates GPU and TPU backends by matching device memory, bandwidth, and interconnect characteristics in the simulator.The configuration uses 40 GB and 80 GB device-memory profiles and corresponding profiled bandwidth parameters.
  • VII. EVALUATION: Simulation accuracy and execution time are compared with Vidur, APEX, TokenSim, and LLMServingSim on configurations executable by at least one baseline.The comparison is restricted because the baseline simulators do not support every serving feature modeled in LLMServingSim 2.0.
  • A. Validation with Real Serving System: 5.14% average throughput error on RTX A6000 and 3.29% on H100 show that simulated time-series throughput tracks real systems across serving modes.The validation includes multi-dense serving, prefix caching, PD disaggregation, and MoE model serving, with variability from batching, arrivals, and phase transitions.
  • A. Validation with Real Serving System: 1.34% average error in total energy consumption shows close power agreement under tensor parallelism degrees 1 and 2.The simulator reproduces active, standby, and idle transitions; higher tensor parallelism raises peak power while shortening execution time narrows power pulses.
  • A. Validation with Real Serving System: Accelerators dominate the seven-component energy breakdown, followed by CPUs and memory, while unused GPUs contribute substantially to the others category.The RTX A6000 server contains four GPUs, so unused GPUs are modeled as constant power consumers.
  • A. Validation with Real Serving System: 0.93% average error in single-instance memory usage and prefix cache hit rate, and 0.41% in multi-instance centralized CPU-cache sharing, demonstrate close temporal agreement.The simulator captures step-wise device-memory growth, per-instance memory growth, and synchronized increases in prefix reuse as requests accumulate.

B. Comparison with Other Simulators

LLMServingSim 2.0 achieves high accuracy across diverse serving configurations, while incurring greater simulation overhead than lightweight simulators because it models serving dynamics, memory behavior, and heterogeneous hardware interactions in detail.

  • Accuracy: 2.43% average error: LLMServingSim 2.0 accurately reproduces normalized throughput, TTFT, and TPOT across single- and multi-instance dense serving configurations.Prior simulators are accurate only for selected metrics and deviate more on others.
  • Accuracy: Under PD disaggregation and MoE serving, several baseline simulators fail to execute or use simplified abstractions that produce large errors.Unsupported configurations are marked unavailable for baseline simulators.
  • Simulation time: Higher simulation overhead than Vidur and TokenSim reflects detailed modeling of serving dynamics and memory behavior.The overhead is presented as a trade-off for accurately capturing system-level interactions.
  • Simulation time: Support for heterogeneous accelerator types and hardware combinations increases simulation time relative to GPU-centric baselines.This overhead follows from LLMServingSim 2.0’s design goal of modeling hardware combinations and system-level interactions.

C. Case Study: Emerging Hardware · VIII. RELATED WORK · IX. CONCLUSION

LLMServingSim 2.0 models emerging TPU and PIM platforms through profile-based operator latency models, reproducing real TPU behavior and exposing system-level PIM effects. Related work covers hardware-centric and serving-level simulators, while the conclusion emphasizes jointly modeling heterogeneous hardware, disaggregation, and runtime dynamics.

  • C. Case Study: Emerging Hardware: LLMServingSim 2.0 incorporates TPU and PIM platforms when specification- or profiling-based per-operator latency models are available.The paper demonstrates this extensibility through emerging-hardware case studies.
  • C. Case Study: Emerging Hardware: The TPU case study uses a TPU-v6e-1 instance, TPU-enabled vLLM, and a TPU-specific profiler, but validation is limited to single-instance dense serving.The limitation follows from the currently stable capabilities of the vLLM-TPU framework.
  • C. Case Study: Emerging Hardware: 3.95% average per-timestep throughput error shows that LLMServingSim 2.0 closely tracks the real TPU system, while full-execution average error falls below 0.3% across evaluated metrics.The results support modeling LLM serving behavior on non-GPU accelerators.
  • C. Case Study: Emerging Hardware: 1.43× higher throughput is achieved by GPU+PIM than GPU-only after prefill, whereas GPU+PIM with sub-batch interleaving has comparable performance to GPU-only.PIM primarily accelerates the memory-intensive decode phase, so gains are limited during the 0–25 s prefill-dominated period.
  • C. Case Study: Emerging Hardware: 14.8% lower J per token is achieved by GPU+PIM, which completes requests faster and consumes less energy than GPU-only and GPU+PIM with SBI.GPU+PIM with SBI consumes more energy because of additional PIM, despite comparable performance to GPU-only.
  • VIII. RELATED WORK: Hardware-centric simulators model accelerator execution, device timing, communication, hardware design, synchronization, or training parallelism, but omit request-driven inference-serving dynamics.The cited systems include LLMCompass, ADOR, ASTRA-sim, and vTrain.
  • VIII. RELATED WORK: System-level simulators evaluate inference scheduling, batching, and deployment configurations, but TokenSim’s simplified memory abstractions overlook bandwidth contention and KV cache movement overhead.Vidur and APEX emphasize latency/throughput prediction through operator profiling and configuration-space exploration for multi-device clusters.
  • IX. CONCLUSION: LLMServingSim 2.0 addresses the challenge of reasoning about interactions among heterogeneous hardware, disaggregated serving architectures, serving techniques, and runtime dynamics.The conclusion frames these interactions as jointly shaping performance, efficiency, and scalability.

APPENDIX · A. Abstract · B. Artifact check-list (meta-information)

LLMServingSim 2.0 is a unified, runtime-driven simulator for heterogeneous and disaggregated LLM serving that models system interactions and serving metrics. Its artifact checklist specifies the implementation environment, evaluation workflow, reported metrics, resources, and availability.

  • A. Abstract: LLMServingSim 2.0 models scheduling, data movement, and contention from system configurations and request traces to estimate throughput and latency breakdowns.It is implemented in Python and C++ and includes power/energy, MoE, prefill-decode disaggregation, memory disaggregation, and multi-tier prefix caching support.
  • B. Artifact check-list (meta-information): The artifact uses gcc/g++ 11.4.0 on Ubuntu 22.04 LTS with Linux kernel 6.8.0-100-generic.Outputs are provided through standard output and CSV files.
  • B. Artifact check-list (meta-information): Experiments reproduce evaluation results for Figures 5-10 using ShareGPT data.The execution workflow runs ./run_all.sh followed by ./compare.sh from the evaluation directory.
  • B. Artifact check-list (meta-information): The artifact reports throughput, latency, prefix hit rate, power, energy, and memory usage as evaluation metrics.Latency includes TTFT, TPOT, ITL, and p99; the listed units are tokens/s, %, W, J, and MB.
  • B. Artifact check-list (meta-information): 15GB of disk space is required, while workflow preparation takes 5 minutes and experiments take 2 hours 30 minutes.These are the checklist’s approximate resource and time estimates.
  • B. Artifact check-list (meta-information): The artifact is publicly available under the Creative Commons Attribution 4.0 International and MIT License licenses.No workflow automation framework is used.
  • APPENDIX: The artifact is archived with DOI 10.5281/zenodo.18879965.The checklist explicitly marks archival status as yes.

C. Description … 5) Models

LLMServingSim 2.0 is evaluated under specified hardware and software dependencies, with request traces generated from ShareGPT and models spanning dense and mixture-of-experts decoder-only transformers. The setup includes x86-64 hardware, a Docker-based software environment, Poisson arrivals, and four evaluated model families.

  • 2) Hardware dependencies: LLMServingSim 2.0 requires an x86-64 architecture for simulation.Simulation time may vary with hardware differences; the authors recommend the Section VI hardware for similar simulation-time results.
  • 2) Hardware dependencies: Hardware differences can affect simulation time.The recommended hardware configuration is the one specified in Section VI.
  • 3) Software dependencies: The simulator was tested on Ubuntu 22.04 LTS with Python 3.10.12 and requires gcc and g++ 11.4.0 or higher.It also requires the software prerequisites of ASTRA-Sim and Chakra.
  • 3) Software dependencies: A prebuilt Docker image and scripts are provided to satisfy the software prerequisites.Reviewers can download and run the image, with further details in Appendix D.
  • 4) Data sets: Request traces are synthesized from the ShareGPT dataset using a Poisson arrival process.The traces provide the request workload for evaluation.
  • 5) Models: The evaluation uses Llama 3.1-8B/70B, Phi-mini MoE, and Mixtral 8×7B.These models follow decoder-only transformer variants and include both dense and MoE designs.

D. Installation • · E. Experiment workflow · F. Evaluation and expected results

LLMServingSim 2.0 is installed by cloning, containerizing, and compiling the repository, then run through a configuration-driven runtime workflow. Its evaluation provides per-figure scripts, aggregate execution, generated artifacts, preserved references, and automated numerical and visual comparisons.

  • D. Installation •: Installation requires cloning the repository with submodules, entering its directory, running Docker, and compiling the submodules.The documented commands are `git clone --recurse-submodules`, `cd LLMServingSim`, `./docker.sh`, and `./compile.sh`.
  • E. Experiment workflow: The simulator takes workload, cluster, and hardware performance-profile configurations as inputs.Hardware profiles are used by each MSG when generating execution graphs.
  • E. Experiment workflow: During its runtime-driven loop, the serving engine routes requests, performs dynamic batching, and generates execution graphs under configured serving policies.The supplied workflow description explicitly connects routing, batching, and graph generation within the runtime loop.
  • F. Evaluation and expected results: The artifact evaluation comprises six experiments corresponding to Fig. 5 through Fig. 10, with one `figure_{i}.sh` script per figure and a `run_all.sh` aggregate script.The scripts are provided in the `evaluation/` folder.
  • F. Evaluation and expected results: Each evaluation script produces logs, results, parsed outputs, and a final figure PDF in its corresponding figure folder.The output folders are `figure_5/` through `figure_10/`, and parsed files contain metrics such as throughput, latency, simulation time, memory, or power.
  • F. Evaluation and expected results: Preserved reference outputs and reference PDFs support numerical comparison through `compare.sh` against archived parsed TSV files.The comparison utility can compare one or more figures or all figures at once.
  • F. Evaluation and expected results: Visual verification compares each generated figure PDF with its corresponding `figure_{i}_ref.pdf`, while per-figure README files document objectives, configurations, datasets, commands, and expected outputs.The evaluation directory also contains utility scripts, archived artifacts, reference PDFs, and README files.

G. Experiment customization … 3) Input parameters

LLMServingSim 2.0 supports experiment customization through cluster configuration files, request-trace datasets, and runtime command-line parameters. These inputs expose hardware, workload, scheduling, feature, and execution controls for configuring simulations.

  • 1) Input configurations: Cluster configuration JSON files define customizable topology and instance settings for the simulated hardware and system.Files are stored under cluster_config/{name}.json and support fields including num_nodes, link_bw, link_latency, hardware, memory, and pd_type.
  • 1) Input configurations: Optional cluster fields extend customization with placement, PIM, power, and CXL memory settings.The optional fields are placement, pim_config, power, and cxl_mem.
  • 2) Input dataset: Request traces use JSONL files in dataset/{name}.jsonl, with each request specifying input_toks, output_toks, arrival_time_ns, and input_tok_ids.Custom traces can be generated with dataset/sharegpt_parser.py or created manually using the same schema.
  • 3) Input parameters: The main runtime accepts --cluster-config, --dataset, and --output options for selecting inputs and recording simulation results.Additional runtime options are documented in README.md.
  • 3) Input parameters: Core parameters control numerical precision, block size, batching limits, and the number of requests through --fp, --block-size, --max-batch, --max-num-batched-tokens, and --num-req.These options configure the core simulation workload and execution settings.
  • 3) Input parameters: Routing and scheduling are configurable with --request-routing-policy, --expert-routing-policy, and --prioritize-prefill.These options appear under routing/scheduling controls.
  • 3) Input parameters: Feature toggles enable prefix caching, prefix sharing, prefix storage, local and attention offloading, sub-batch interleaving, and attention prediction.The corresponding options are --enable-prefix-caching, --enable-prefix-sharing, --prefix-storage, --enable-local-offloading, --enable-attn-offloading, --enable-sub-batch-interleaving, and --enable-attn-prediction.
  • 3) Input parameters: Run control and logging use --gen, --log-interval, --log-level, and --network-backend.These options provide controls for generation, logging behavior, and the network backend.

4) Evaluation-script customization · H. Notes

Evaluation scripts expose configurable inputs for cluster configurations, datasets, workloads, and runtime execution, with additional directory-specific guidance available in README.md files.

  • 4) Evaluation-script customization: Each per-figure script, named figure_{i}.sh, declares key inputs near its top.These inputs include cluster-configuration paths, dataset paths, and runtime options passed to main.py.
  • 4) Evaluation-script customization: Users can customize configuration files through the input parameters listed in Appendix G3.
  • 4) Evaluation-script customization: Users can customize workload traces through the input parameters listed in Appendix G3.
  • 4) Evaluation-script customization: Users can customize execution settings through the input parameters listed in Appendix G3.
  • H. Notes: Additional information is available in the README.md file of each directory.
  • H. Notes: The README.md guidance is organized by directory.
Loading 2602.23036v2…