ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production

1 Introduction

• Prior workload analyses lack comprehensive production-scale coverage, leaving real-world serving patterns insufficiently characterized.The status quo includes adapted workloads from other domains and analyses limited to particular optimizations or patterns.Source: §1 · p. 1
• The study characterizes language, multimodal, and reasoning workloads, revealing distinct workload patterns in each area.It analyzes multimodal load variance and request heterogeneity, and reasoning-length distributions and multi-turn arrival patterns.Source: §1 · pp. 1–2
• Language-model arrivals are bursty beyond any single stochastic process, while input/output length parameters vary over time.Client decomposition attributes much of this nondeterminism to several top clients, while most clients remain stable and predictable.Source: §1 · pp. 1–2
• Production workloads can be unavailable because of privacy concerns, while published workloads may be limited in scale and subject to workload churn.ServeGen models workloads per client based on the study's findings.Source: §1 · p. 2
• ServeGen generates workloads that align better with real workloads than NAIVE and supports benchmarking through production case studies.The case studies cover instance provisioning and PD disaggregation.Source: §1 · p. 2

2.1 LLM Basics

• LLM requests pass through prefill, which processes prompt tokens to produce the first output token, and decoding, which generates subsequent tokens sequentially.Requests are commonly batched, so arrival patterns and input/output lengths affect batching and computational load.Source: §2.1 · p. 2
• Multimodal inference processes inputs through downloading, normalization, and modality-specific encoding before combining them with text embeddings.The subsequent inference proceeds like basic LLM serving, and multimodal data distributions affect performance.Source: §2.1 · pp. 2–3
• Reasoning-model outputs contain reason tokens for test-time computation followed by answer tokens, changing workload statistics such as output length.These workloads may also enable new optimizations.Source: §2.1 · p. 3

2.2 LLM Serving Workloads

• Prior workload characterizations lack scale and leave patterns uncharacterized, motivating a more extensive production study and the release of ServeGen.The framework generates realistic serving workloads to support future research.Source: §2.2 · p. 3
• The workload source is Alibaba Cloud Model Studio, a worldwide service platform with over 200 foundation models, thousands of fine-tuned models, and millions of daily requests.The study uses sanitized request metadata spanning four months, 12 models, billions of requests, and multiple datacenter geolocations.Source: §2.2 · p. 3

3.1 Request Arrival Pattern

• Arrivals are often bursty (CV > 1), with no universally best inter-arrival model, while rates and burstiness shift across workloads and time.Gamma fits M-large, Weibull fits M-mid, and Exponential can fit M-small; some workloads also show diurnal rate changes.Source: §3.1 · pp. 3–4, Figure 1 · p. 4, Figure 2 · p. 4

3.2 Input and Output Length Distribution

• Input lengths fit Pareto and Log-normal mixtures in general workloads, outputs mostly fit Exponential distributions, and input/output length correlation is weak.Task-specific input fits are less accurate when domain-specific prompts or templates bias the distribution.Source: §3.2 · p. 4, Figure 3 · p. 5
• Input and output length distributions shift independently over time, with average-length shifts reaching 1.63× for input and 1.46× for output.M-mid input length rises 13% while output length drops 18% from midnight to afternoon, changing load across prefill and decoding.Source: §3.2 · p. 5

3.3 Client Decomposition

• Client rates are highly skewed: 29 of 2,412 clients account for 90% of requests, while client burstiness and request lengths vary widely.Top clients are comparatively stable in burstiness and lengths when considered individually.Source: §3.3 · p. 5, Figure 5 · p. 6
• A few top clients, despite heterogeneous behavior, largely explain workload shifts through their rate fluctuations.The decomposition links Client A’s rate peaks to bursts and changing average input lengths in M-small.Source: §3.3 · pp. 5–6

4.1 Modality Load Variance

• Image, audio, and video inputs have irregular length distributions and weak correlation with text, yielding variable encoder load; mm-omni shows more complex load variation.In mm-image, average image length varies by up to 19% over one day, and modality load can shift independently of text load.Source: §4.1 · p. 6, Figure 7 · p. 7

4.2 Request Heterogeneity

• Multimodal requests range from text-heavy to multimodal-heavy, and their download, normalization, and encoding stages can prolong TTFT.Half of mm-image requests spend 75% of TTFT before LLM prefilling; long-tailed encoder times can also queue text-heavy requests.Source: §4.2 · pp. 6–7, Figure 9 · p. 7

4.3 Mulitmodal Client Decomposition

• The 1,036 mm-image clients vary in rate, burstiness, image lengths, and image-to-input ratios, with staircase-like image-data CDFs suggesting text-heavy or multimodal-heavy clients.Client B consistently sends images of around 1,200 tokens each across the observed 24 hours.Source: §4.3 · p. 7
• Top-client behavior connects client heterogeneity to aggregate patterns: Client B’s rate ramp coincides with a surge of image-heavy requests and increased image load.Source: §4.3 · p. 7

5.1 Understanding Reason & Answer Lengths

• Reasoning outputs are longer and more variable than non-reasoning outputs; reason lengths average 4× answer lengths and contribute more to output-length shifts.Figure 13 characterizes deepseek-r1 request lengths and separates reason from answer tokens.Source: Figure 13 · p. 8, §5.1 · p. 8
• Reason and answer lengths correlate more clearly than in Figure 4, while their per-request ratio is bimodal, reflecting more-complete and more-concise answer patterns.The bimodal ratio may offer an opportunity for future serving optimizations.Source: §5.1 · p. 8, Figure 13 · p. 8

5.2 Arrival and Multi-Turn Patterns

• Deepseek-r1 and deepqwen-r1 arrivals are non-bursty despite diurnal rate shifts, and recurring multi-turn conversations affect reasoning-workload arrival patterns.Their inter-arrival times fit an Exponential distribution reasonably well, roughly modeling Poisson processes.Source: §5.2 · pp. 8–9, Figure 14 · p. 9
• In a 12-hour deepseek-r1 window, 188,986 multi-turn requests formed 57,205 conversations averaging 3.5 turns; preserving inter-turn times produced more stable upsampled arrivals than naive scaling.The analysis identifies 188,986 multi-turn requests among 1,964,415 total requests; inter-turn times concentrate around 100 seconds with a long tail.Source: §5.2 · p. 9, Figure 15 · p. 9, Figure 16 · p. 9

5.3 Decomposition of Reasoning Workloads

• Among 25,913 deepseek-r1 clients, the top 10 account for half of requests, while clients are less bursty and their output lengths retain a bimodal pattern.The day-and-night shift in answer-length ratio is attributed to client-rate fluctuations.Source: Figure 17 · p. 9, §5.3 · p. 9

6.1 ServeGen Framework

• ServeGen samples requests per client and aggregates them; client traces and datasets can be parameterized or supplied as samples, and the framework can preserve conversation histories through conversation-aware mocking.Users provide a client count and total arrival rate; client behaviors can come from a preconfigured pool or user-specified clients.Source: Figure 18 · p. 10, §6.1 · p. 10
• ServeGen uses prior workload findings to model changing client and total rates over time, sample heterogeneous clients, and configure parameterized real-world client behaviors.Users may also use selected findings to sanity-check generated-workload statistics.Source: §6.1 · p. 10

6.2 Generation Accuracy

• ServeGen workloads match actual workload characteristics better than NAIVE in every generation-accuracy case, while NAIVE captures rate variability and rate–data correlations poorly.The comparison targets variability in request-data distributions over time, including stable and variable rate periods.Source: §6.2 · p. 10, Figure 19 · p. 11

6.3 Use Case #1: Instance Provisioning

• For instance provisioning, the study benchmarks a single instance across P99 TTFT and TBT SLO targets, derives instance counts, and checks them against actual M-large traffic.Figure 20 reports provisioned instance counts and over- or under-provisioning percentages.Source: Figure 20 · p. 11, §6.3 · p. 11

6.4 Use Case #2: PD-Disaggregation

• In two of three SLO cases, 2P6D is the best PD-disaggregation configuration under ServeGen, although NAIVE benchmarking reports it as worse.The discrepancy is attributed as likely due to intricate long-tailed requests in ServeGen, producing more extreme load imbalance and greater decoding-instance demand.Source: §6.4 · pp. 11–12, Figure 21 · p. 12

7 Discussion

• Multimodal analysis finds that preprocessing contributes substantially to TTFT, motivating full-stack optimization such as independently scaling modality encoders.The preprocessing stages named are downloading, normalization, and encoding.Source: §7 · p. 12
• The non-bursty arrival pattern of multi-turn reasoning requests may inform short-term workload predictability in conversational scenarios.The paper presents this as a potential research direction.Source: §7 · p. 12
• The workload findings have informed Alibaba Cloud Model Studio teams working on inference-engine optimization, resource planning, and request scheduling.Source: §7 · p. 12
• ServeGen does not yet characterize plugin-call applications or prefix caching, and its workload analysis is not long enough for longitudinal studies.Prefix-caching characterization is constrained by confidentiality obligations that prevent access to request content.Source: §7 · p. 12

8 Related Work

• Earlier LLM-serving workload studies characterize language and image-to-text workloads or examine narrower aspects such as burstiness, computational load, prefix sharing, and energy efficiency.Source: §8 · p. 12
• ServeGen is presented as broader in scale and scope than prior LLM-serving workload characterization, while complementing serving simulators for realistic end-to-end performance modeling.Source: §8 · p. 12
• Generic workload generators model historical workloads, BurstGPT models LLM-serving burstiness with a parameterized Gamma process, and many serving studies use NAIVE trace-and-dataset resampling.Source: §8 · p. 12

• The characterization covers real-world serving workloads for language, multimodal, and reasoning models.Source: §9 · p. 12
• ServeGen generates realistic serving workloads by composing them on a per-client basis.Source: §9 · p. 12
• The paper demonstrates ServeGen's benefits through case studies on instance provisioning and PD disaggregation.Source: §9 · p. 12