PaperlayerView exact revision on arXiv

Source-linked AI summary

GAUSS: A Workload-Aware Analytical and Monte-Carlo Simulator for LLM Inference Serving

Srikant Bharadwaj, Ankur Mallick, Spyridon Mastorakis, Anjaly Parayil, Dhruv Deshmukh, Victor Ruehle

arXiv:2610.37989v1cs.AI

TL;DR

LLM serving has workload-dependent prefill, decode, and queueing behavior, so a single throughput figure does not capture user-visible latency targets. GAUSS combines calibrated phase-specific latency models with stochastic queueing and Monte Carlo sampling to predict latency distributions and sustainable request rates for workloads and configurations, including ones not yet deployed.

  • Problem

    LLM serving combines distinct prefill and decode phases with continuous batching and workload variation, so average throughput alone cannot answer latency-SLO and capacity questions.

    Source: §1 · p. 1

  • Method

    GAUSS combines weighted request buckets and stochastic arrivals with calibrated prefill and decode latency surfaces, queueing, and ensemble Monte Carlo estimation.

    Source: §1 · p. 2, §3 · p. 5, §3.1 · p. 5

  • Results

    GAUSS predicts TTFT and TBT distributions and sustainable request rates for workloads, policies, and hardware that have not yet been deployed.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The model connects workload composition, arrival burstiness, and engine calibration to the user-visible metrics operators manage across candidate configurations.

    Source: §1 · p. 3

    Queueing is modeled as one M/G/1 server whose job is a request’s full chunked prefill served first-in, first-out.

    Source: Modeling assumptions · p. 7

Abstract

from arXiv · show

Large language model (LLM) inference has become an operational workload that fleets must provision, monitor, and bill against latency service-level objectives (SLOs). A single throughput number cannot answer the questions operators actually face, because generative serving couples two phases with very different service characteristics, continuous batching reshapes the effective service process, and user-visible tail latency can degrade long before aggregate throughput saturates. We present GAUSS, a workload-aware analytical and Monte-Carlo simulator that predicts the full distributions of time-to-first-token (TTFT) and time-between-tokens (TBT), together with the sustainable request rate, for a given workload, engine, and accelerator. GAUSS pairs calibrated, phase-specific latency surfaces for prefill and decode with a Lindley-recurrence model of queueing under stochastic arrivals and estimates performance by ensemble Monte-Carlo sampling rather than by replaying a single trace. This lets GAUSS reason about workloads, routing policies, and hardware that have never been deployed, and to capture the latency tails that trace-driven simulators systematically under-sample. We describe the model and show how it supports operating-point selection, traffic segmentation, and shortest-remaining-processing-time scheduling capacity questions that trace-driven simulators cannot address.

1. Introduction

LLM serving couples distinct prefill and decode phases with workload-dependent batching and bursty arrivals, so average throughput can conceal latency-SLO violations. GAUSS combines workload-aware, calibrated service models with stochastic queueing and ensemble Monte Carlo to estimate latency distributions and capacity for candidate deployments.

1. Introduction

  • Workload heterogeneity and bursty arrivals interact with engine-specific service times and queueing, making capacity, SLO, utilization, and cost questions workload-dependent.Traffic varies in prompt, cached-prefix, and generated lengths; average throughput alone can mask first-token or inter-token SLO violations.Source: §1 · p. 1
  • An analytical model evaluates service time as a calibrated function rather than a logged trace value, enabling ensemble realizations and compact sweeps across candidate workloads and engines.These capabilities address trace replay’s limits in sampling latency tails and assessing unrun configurations.Source: §1 · p. 2
  • Because service time depends on request shape and engine calibration, GAUSS predicts latency and capacity for workloads, policies, and hardware absent from recorded traces.Trace-driven simulation is bound to the workload and configuration captured in its trace.Source: §1 · p. 2
  • GAUSS combines weighted request buckets and stochastic arrivals with calibrated, phase-specific prefill and decode latency surfaces to model candidate offered loads.The surfaces expose how context length, cached context, and batch size shape service time.Source: §1 · p. 2
  • GAUSS estimates TTFT and TBT distributions, not just average throughput, by composing prefill work, decode-batch cost, and Lindley-style queue delay.Ensemble Monte Carlo estimates latency percentiles, including the tails relevant to SLOs.Source: §1 · p. 2
  • GAUSS sweeps candidate request rates to select an operating point before tail latency degrades sharply, supporting capacity planning, SLO exploration, and utilization analysis.Its unifying contribution connects workload composition, arrival burstiness, and engine calibration to TTFT, TBT, and sustainable request rate for configurations that need not yet exist.Source: §1 · pp. 2–3

2. Background

LLM inference performance reflects accelerator memory and compute behavior, batching mechanisms, and queueing—not a single fixed service time. GAUSS abstracts these effects into calibrated latency models and combines stochastic workload modeling with queue-backlog recurrence to predict latency and capacity.

2.1 GPU Architecture and the Two Phases of Inference

  • Prefill is compute-bound, while decode is memory-bound; context length, batch size, and KV-cache capacity shape their respective costs.Prefill processes prompt tokens in parallel; decode reads weights and the full KV cache at each token step.Source: §2.1 · p. 3

2.2 Batched Inference Algorithms for LLMs

  • Continuous batching, paged KV-cache management, and chunked prefill address batch utilization, memory occupancy, and decode latency, respectively.GAUSS models the performance consequences of these mechanisms rather than proposing a new serving mechanism.Source: §2.2 · pp. 3–4

2.3 Positioning Among Inference Simulators and Capacity Models

  • Trace-driven discrete-event simulators such as Vidur support high-fidelity study of specific schedulers and runtimes, but are heavier to sweep across workloads and rates [5].Earlier profile-driven frameworks use stage profiles, latency SLOs, and arrival assumptions, but model fixed-cost stages rather than context- and batch-dependent LLM service.Source: §2.3 · p. 4
  • GAUSS combines calibrated phase-specific latency surfaces, weighted workload buckets, stochastic arrivals, and queue-backlog recurrence to predict TTFT, TBT, and capacity.Its analytical, probabilistic approach estimates ensemble distributions, captures tails, and extrapolates to workloads, policies, and hardware absent from traces.Source: §2.3 · p. 4

3. Design of the Simulator

GAUSS converts workload descriptions and engine calibration into latency distributions by modeling request service time and queueing. It represents traffic with weighted token-count buckets and derives prefill cost analytically from the uncached prompt and attended context.

3. Design of the Simulator

  • GAUSS turns a workload description and engine calibration into latency distributions by modeling individual-request service time, queueing delay, and their composition into TTFT and TBT.The described model covers decoder-only models at a single serving stage.Source: §3 · p. 5

3.1 Workload and Arrival Model

  • Each request bucket specifies input, cached-prefix, and output token counts with a probability weight; arrivals follow a Poisson process with mean rate λ.The paper reports that busy-period bucket distributions are stable across days when a shared system prompt is cached and reused.Source: §3.1 · p. 5

3.2 Prefill Service Time

  • Summing chunk costs yields a closed-form prefill compute that is quadratic in the number of chunks because attended context grows with chunk index.Each chunk attends to the cached prefix plus previously processed chunks; the per-chunk latency is affine in attended context, with engine coefficients a and b.Source: §3.2 · p. 5

3.3 Decode-Batch Latency

GAUSS models decode-batch latency as an affine function of batch size and average context per request. Its terms represent KV-state reading and the marginal cost of widening the batch, with calibrated coefficients that may vary across batch-size regimes.

3.3 Decode-Batch Latency

  • Decode-step latency is modeled as d(B, C) = p + q·(C / B) + r·B, an affine surface in batch size and aggregate resident context.The model uses calibrated coefficients p, q, and r.Source: §3.3 · p. 5
  • The C/B term captures each request’s KV-state reading cost, while the linear B term captures the marginal cost of widening the batch.In practice, coefficients are piecewise across batch-size regimes; the single affine form is used for clarity.Source: §3.3 · p. 5
  • Decode-batch latency also contributes to prefill service time because chunked prefill folds each chunk into a concurrent decode batch.Source: §3.3 · p. 6

3.4 Service Time Under Concurrent Prefill and Decode

Concurrent chunked prefill adds the cost of its own chunks and the decode-batch work during the steps those chunks occupy. The same interleaving affects per-request TBT, with smaller chunks bounding the added latency at a throughput cost.

3.4 Service Time Under Concurrent Prefill and Decode

  • A request’s prefill service time combines its chunked-prefill compute with the concurrent decode-batch cost of the steps its chunks occupy.The model expresses this as S(s, c) = (a·n + b·σ + n·d̄) / η, where d̄ is concurrent-batch decode latency and η converts modeled work into wall-clock time.Source: §3.4 · p. 6
  • During steady decoding, each step costs d(B, C), and inserting a prefill chunk inflates that step by the chunk’s marginal cost.Source: §3.4 · p. 6
  • Averaging over generated tokens gives per-request TBT; keeping chunk size Δ small bounds the inflation while trading off throughput.Source: §3.4 · p. 6

3.5 Service Rate and the Operating Point

GAUSS averages request service times across workload buckets to compute the service rate and sustainable request-rate ceiling.

3.5 Service Rate and the Operating Point

  • GAUSS averages service time over workload buckets to obtain expected service time and its corresponding service rate μ.The averaging weights reflect the workload mix.Source: §3.5 · p. 6
  • The maximum sustainable request rate is λ* = μ.This ceiling follows from the service rate calculated for the workload.Source: §3.5 · p. 6
  • The system is stable only while load ρ = λ/μ remains below one.Source: §3.5 · p. 6

3.6 Queueing Delay via a Lindley Recurrence

GAUSS models prefill admission as a single-server FIFO queue and uses Monte Carlo sampling to estimate the resulting queue-delay distribution.

3.6 Queueing Delay via a Lindley Recurrence

  • Prefill admission is modeled as a single-server, first-in, first-out queue with general service times (M/G/1).Request service times are sampled from the workload, while inter-arrival times are sampled exponentially at rate λ.Source: §3.6 · p. 6
  • GAUSS evaluates Lindley’s recurrence over many sampled service and inter-arrival sequences instead of solving for a stationary distribution in closed form.Source: §3.6 · p. 6
  • The recurrence sampling yields the full ensemble distribution of queue delay W rather than only its mean.Source: §3.6 · p. 6

3.7 Composing TTFT and TBT

GAUSS composes queue delay with prefill service to estimate TTFT, while decode-batch latency and interleaved prefill determine TBT; sweeping load traces percentile changes.

3.7 Composing TTFT and TBT

  • TTFT includes both the time waiting in the queue and the time to finish prefill.Source: §3.7 · p. 6
  • Sampling queue delay W and prefill service S and adding them produces the steady-state TTFT distribution.Source: §3.7 · p. 6
  • TBT combines decode-batch latency with interleaved prefill inflation averaged over the request’s generated tokens.Source: §3.7 · pp. 6–7
  • Sweeping arrival rate λ and workload mix shows how TTFT and TBT percentiles move and locates the rate where the tail degrades sharply.Source: §3.7 · p. 7

3.8 Batch Size, Backlog, and the Estimation Procedure

GAUSS determines steady-state batch size from offered-load time budgets, then Monte Carlo estimates latency distributions and service-rate outputs.

3.8 Batch Size, Backlog, and the Estimation Procedure

  • The steady-state batch size is determined by work that fits within the per-request time budget set by offered load.Balancing prefill and decode work against that budget yields a quadratic whose root gives the operating batch size in closed form.Source: §3.8 · p. 7
  • Algorithm 1 takes workload buckets, arrival rate, engine coefficients, and sample counts, and outputs TTFT and TBT distributions, service rate μ, and sustainable rate λ*.Source: §3.8 · p. 7
  • The estimation procedure computes bucket service times and μ, sweeps Lindley waiting times, samples TTFT, and forms decode batches for TBT.It returns percentiles for TTFT and TBT together with μ and λ*.Source: §3.8 · p. 7

Modeling assumptions

GAUSS makes capacity predictions under a stable peak workload, Poisson arrivals, single-server FIFO prefill admission, and calibrated latency surfaces.

  • The workload distribution is assumed stable over the planning window, based on cached-system-prompt concentration.Source: Modeling assumptions · p. 7
  • Requests are assumed to arrive as a Poisson process with mean rate λ and exponentially distributed inter-arrival times.Source: Modeling assumptions · p. 7
  • Prefill admission is modeled as one M/G/1 server processing each request’s full chunked prefill in first-in, first-out order.Source: Modeling assumptions · p. 7
  • Per-chunk prefill latency is affine in attended context, while decode-batch latency is piecewise affine in batch size and average context.Coefficients are fit separately for each engine and accelerator.Source: Modeling assumptions · p. 7

4. Example Use Cases

GAUSS applies analytical service-time and latency distributions to capacity planning, traffic segmentation, SRPT scheduling, and fleet-level analyses. Its ensemble estimates support tail-aware operating decisions and predictions for configurations not yet deployed.

4. Example Use Cases

  • The example use cases apply GAUSS’s analytical predictions to operational questions that trace-driven simulation cannot answer.The section covers three concrete tasks and a broader class of ensemble-based fleet applications.Source: §4 · p. 8

4.1 Operating-Point and Capacity Calculation

  • Operating just below the service-rate knee captures most achievable utilization while keeping tail latency within the SLO.The gap between the operating point and the knee is headroom for routing and provisioning.Source: §4.1 · p. 8

4.2 Traffic Segmentation

  • GAUSS computes service-time moments for candidate traffic segments, allowing it to search analytically for the boundary that minimizes overall delay.It can evaluate partitions of unobserved workloads without rerunning a trace for each candidate boundary.Source: §4.2 · p. 8

4.3 Shortest-Remaining-Processing-Time Scheduling

  • GAUSS supplies the remaining-time function needed by SRPT and can quantify its potential tail-latency reduction before deployment.For chunked prefill, remaining work is computed from the cost of the request’s unprocessed chunks.Source: §4.3 · pp. 8–9

4.4 Future Use Cases: Ensemble Estimation for Fleet Operations

  • Ensemble sampling estimates full latency distributions, including P95 and P99, more reliably than a single trace trajectory under high variability and burstiness.Source: §4.4 · p. 9
Loading 2610.37989v1…