PaperlayerView exact revision on arXiv

Source-linked AI summary

A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing

William Nixon, Jon Durbin, Florian Standhartinger, Haryadi S. Gunawi, Juncheng Yang

arXiv:2608.13573v1cs.AI

TL;DR

Existing LLM serving traces provide limited long-term, request-level visibility into production workload evolution and user–model interactions. This paper analyzes and releases a one-year unsampled production trace, finding that workloads are non-stationary and that prefix-cache locality trades off against load balancing.

  • Problem

    Existing public LLM serving artifacts are often sampled, omit user identifiers, or use synthetic generators, limiting long-term and request-level workload analysis.

  • Method

    The paper collects, characterizes, and releases a one-year unsampled production trace spanning users, models, requests, serving instances, token counts, latency, and cache reuse.

  • Results

    Production workloads are non-stationary, while prefix-cache locality improves reuse but increases load imbalance and cache replication.

  • Takeaways & Limitations

    Trace-driven serving designs should account for temporal evolution, workload structure, and cache-aware routing.

  • Takeaways & Limitations

    The session-reconstruction simulator estimates workload-intrinsic reuse rather than reuse induced by production routing locality.

Abstract

from arXiv · show

Large Language Model (LLM) serving has become a critical cloud workload, and realistic traces are essential for motivating and benchmarking serving systems. However, existing LLM serving workload studies remain limited in scale and scope. They often observe short time periods and provide limited visibility into how users interact with models in production. As a result, they do not fully capture how LLM serving workloads evolve over time or how user-model interactions shape production traffic. In this work, we further the understanding of real-world LLM serving workloads through both a global characterization and a longitudinal study of a one-year production trace from Chutes. Unlike prior studies, our trace captures full production behavior across many models and users, including both popular and long-tail models. We analyze the workload from aggregate, temporal, model-level, and user-level perspectives, revealing workload evolution and user-model structure that are typically hidden behind aggregate views. To support future research, we will release the full one-year trace with the paper, enabling downstream studies of production behavior without relying on sampled or synthetically generated workloads.

1 Introduction

The paper addresses gaps in production LLM-serving workload studies through a one-year, unsampled request-level trace and analyses of workload evolution, model and user interactions, caching, and load balancing. Its findings show that production behavior is non-stationary and reveals structure hidden by aggregate views.

  • Introduction: The authors collect and release a one-year, unsampled request-level production trace spanning billions of requests, thousands of models, anonymized users and instances, token counts, latency, and prefix-cache reuse.The trace provides broad coverage of popular and long-tail models and supports analyses difficult to perform from aggregate statistics alone.
  • Introduction: LLM serving workloads are non-stationary over yearly timescales, shifting toward longer inputs and shorter outputs.These shifts change relative pressure on prefill and decode, making short measurement windows insufficient for capacity planning.
  • Introduction: Many long-tail models receive sparse traffic with long inter-arrival times, while consecutive intervals often remain correlated.This structure creates opportunities to colocate or multiplex low-traffic models on shared GPUs rather than operate each as an independent always-on service.
  • Introduction: Users interact with models heterogeneously: some repeatedly use one model, whereas others switch across many models.These patterns are hidden in aggregate statistics and suggest opportunities for user-aware scheduling, placement, and caching.
  • Introduction: Prefix-cache token hit ratios are often bimodal, with most requests seeing either near-zero or near-complete reuse.Reuse is temporally concentrated within short inter-arrival windows, and simple FIFO and LRU can outperform state-of-the-art policies designed for web and storage workloads.
  • Introduction: Routing related requests to the same host improves prefix-cache locality and hit ratio but increases load imbalance and cache replication.This tradeoff motivates joint treatment of routing, placement, and caching in scheduler design.

2 Background

LLM inference has distinct prefill and decode phases whose costs depend on input and output tokens, making request timing, token counts, caching, and user-model metadata central to workload analysis. Existing public artifacts limit independent study of long-term evolution and user–model interactions, motivating a one-year, fine-grained production trace.

  • 2 Background: LLM inference comprises prefill, which builds the request’s KV cache, and autoregressive decode, which uses cached previous tokens.Input length mainly affects prefill cost, while output length affects decode duration.
  • 2 Background: Request arrivals shape batching and queueing; input and output tokens create prefill and decode pressure; cached tokens quantify avoidable prompt work.The trace therefore records request timing, model and user identifiers, token counts, latency, TTFT, and cached-token information.
  • 2 Background: Existing public artifacts are sampled, omit fields such as user identifiers, or provide synthetic generators instead of full request-level production traces.These limitations hinder independent analysis of long-term workload evolution, user–model interactions, and faithful inter-arrival behavior.
  • 2 Background: 6.12 billion requests across 9,174 models are included in a one-year production trace retaining anonymized user, serving-instance, TTFT, and cached-token information.This combination enables longitudinal and fine-grained analysis of workload evolution, user–model interactions, prefix reuse, and routing-related behavior.

3 Production Trace Analysis

The one-year production trace spans billions of requests across hundreds of thousands of users and thousands of public and private models, with request- and serving-side signals enabling detailed workload analysis. Demand is non-stationary across time, models, APIs, token shapes, and latency phases, requiring workload-aware analysis and optimization.

  • Trace scope: 6.12 billion requests from 314,970 users across 9,174 models—including 3,922 public and 5,252 private models—form the one-year production trace.The trace covers both platform-hosted and user-deployed models, including restricted-access user-fine-tuned variants.
  • Trace scope: Per-request records combine timestamps, endpoints, models, users, token counts, handling instances, latency, TTFT, and cached-token counts.These fields support analysis of prefix-cache reuse and instance-level load balancing.
  • Temporal workload structure: Daily request and token volumes, active-model counts, and API mix change over the year, while traffic remains highest during UTC afternoons and evenings, especially weekdays.Active models rise from fewer than 100 to more than 400 at peak, and demand is lower overnight and on weekends.
  • Request and model profiles: Requests are often prompt-heavy, with inputs spanning roughly 10^2 to 10^5 tokens, outputs mostly between 10^1 and 10^3 tokens, and outputs usually shorter than inputs.Model profiles differ: MiniMax-M2.5 is input-heavy, DeepSeek R1 has heavier output behavior, and DeepSeek V3.2 lies between them.
  • Latency structure: End-to-end latency spans sub-second requests to tens of seconds and is usually decode-dominated, but models can be latency-bound by long outputs or long prompts.DeepSeek R1 has the longest latency, MiniMax-M2.5 has a larger TTFT share, and output length drives duration while input length drives TTFT.

4 Users and Models

Users and models with similar request volumes exhibit substantially different access breadth, token shapes, and temporal behaviors. These patterns reveal serving-relevant structure, including persistent burstiness and opportunities for caching, routing, placement, and capacity planning.

  • Users and Models: Models with similar request counts can differ by orders of magnitude in user count, while most heavily used models receive most traffic from relatively few users.Some high-traffic models are broadly used, whereas others serve a much smaller user base, creating different locality characteristics.
  • Users and Models: Most users issue few requests to a small model set, while a sparse tail of high-volume users accesses many models.These broad-access power users may represent automated applications, proxy services, model routers, or agentic workflows.
  • Users and Models: Most users and models are input-heavy at the median, but wide variation in input and output lengths makes request count an incomplete proxy for serving cost.Users and models with similar request volumes can impose different costs depending on traffic shape.
  • Temporal Access Patterns: High-volume users differ temporally: some repeatedly access a small stable model set, while others briefly expand to many models after focusing on a few.Popular models likewise show distinct patterns, including persistent user activity, periodic requests, gradual new-user arrivals, and correlated access across accounts.
  • Serving Implications: Distinct user- and model-level temporal patterns can inform caching, routing, model placement, and capacity provisioning.These histories provide coarser, more stable signals than fine-grained block or individual-job behavior.
  • Burstiness: Most models have inter-arrival-time CV > 1 across low- and high-rate groups, and many bursty models exhibit positive autocorrelation.Busy and quiet periods therefore tend to persist, creating opportunities for autoscaling, colocation, and multiplexing.

5 One-Year Workload Evolution

Over one year, dominant models turn over and traffic spreads across a broader model mix. Input lengths remain broadly stable, while outputs shorten and newer user cohorts submit larger, more variable prompts.

  • 5.1 Model usage: Dominant models turn over substantially: early traffic is led by DeepSeek models, while Qwen3-32B and DeepSeek-V3.2 become more prominent later.The same trend appears in token-weighted traffic.
  • 5.1 Model usage: Traffic becomes more diverse over time as the “Others” share grows, spreading requests and tokens across more non-dominant models.This changing model mix can make optimizations tuned to earlier leaders less suitable later.
  • 5.2 Token shape: Input lengths remain broadly stable, with monthly medians in the thousands of tokens, while median output length falls from a few hundred to below one hundred tokens.Output-length variation also increases toward the end of the trace.
  • 5.2 Token shape: Newer user cohorts submit longer prompts, while their median output change is less pronounced and token-length variation widens for both inputs and outputs.Later cohorts therefore shift toward larger and more variable token lengths.

6 Prefix Caching

Prefix-cache reuse is common but uneven: request-level reuse is bimodal, cacheability varies across models and users, and repeat arrivals are concentrated at short timescales. Simulations show that recency-driven FIFO and LRU are highly competitive, although workload-dependent gaps to Belady remain.

  • Production cache behavior: Prefix-cache reuse is bimodal across requests, with most requests receiving little reuse or nearly their entire input prefix.The production analysis uses logged cached-token counts from the final two months, capturing reuse realized by the deployed system.
  • Production cache behavior: Cacheability varies sharply by model and user: DeepSeek models often have zero hits, MiniMax-M2.5 has broader higher-hit distributions, and a smaller user group achieves high ratios.Models with modest request volumes can still achieve high token hit ratios when traffic and contexts are concentrated.
  • Arrival locality: 99% of requests from the same user-model pair return within 15 minutes, with roughly half arriving within 0.1 second and around 80% within 10 seconds.Per-user inter-arrival times remain on the order of minutes for most of the trace but decline toward early 2026, while recurrence varies substantially by model.
  • Eviction algorithms: LRU and FIFO often match or exceed more complex state-of-the-art eviction algorithms because prefix reuse is strongly recency-driven.ARC performs substantially worse than LRU at intermediate cache sizes, especially on DeepSeek V3.2.
  • Eviction algorithms: MiniMax-M2.5 approaches the Belady oracle at moderate cache sizes, whereas DeepSeek V3.2 remains below it even at large capacities.This workload dependence indicates that MiniMax-M2.5 is more cache-friendly while DeepSeek V3.2 is more cache-constrained, leaving room for LLM-specific cache designs.

7 Load Balancing

Load balancing in LLM serving follows token work rather than request volume, yet shrinking fleets become more imbalanced and high load forces traffic to spread across replicas. Cache-aware routing improves reuse and limits KV duplication with modest imbalance, exposing a fundamental tradeoff between cache locality and load balance.

  • Aggregate load: Replica count tracks total token work rather than request arrivals, while per-hour max/mean token-load imbalance worsens as the serving pool shrinks.Request rates remain bursty but roughly stationary as active instances decline with input-plus-output token volume.
  • Routing tradeoff: Routing first considers the few least-loaded instances, then selects the one with the highest expected prefix-cache hits; low load concentrates sessions, whereas high load spreads them across replicas.Under high load, spreading avoids overloading one replica but duplicates KV state; under low load, concentration preserves locality.
  • Routing tradeoff: A 100K-token MiniMax-M2.5 context occupies roughly 27 GB of KV state and needs about 2.7 s to transfer over a 10 GB/s link in one direction.The estimate excludes contention, coordination, and any additional transfer to the destination worker; reuse often occurs within seconds, limiting transfer’s latency benefit.
  • Cache-aware routing: Cache-aware routing improves reuse and reduces cross-instance KV duplication with modest imbalance, whereas cache-blind routing increasingly replicates sessions as fleet size grows.Larger caches strengthen affinity and reduce session spread, but cannot recover reuse when routing scatters session state across replicas; cache_first pays higher imbalance than round_robin and load_first.

8 Related Work

Prior LLM serving research improves batching, scheduling, memory management, prefill–decode separation, and KV-cache reuse, while these optimizations require realistic datasets for faithful evaluation. Existing evaluations often use benchmark or application-specific workloads or short, partially released production traces.

  • LLM Inference Optimization: LLM serving optimization spans batching, scheduling, memory management, prefill–decode separation, and KV-cache reuse.Orca, vLLM, and Sarathi-Serve improve continuous batching, KV-cache memory management, and prefill scheduling; DistServe, Splitwise, and Mooncake disaggregate serving components.
  • LLM Inference Optimization: Disaggregated serving and cache-aware routing target serving efficiency through prefill, decode, KV-cache management, temporal locality, and prefix reuse.The passage identifies disaggregated serving and direct cache-reuse work as major optimization directions and links cache-aware routing to temporal locality and prefix reuse.
  • Evaluation Workloads: Faithful evaluation depends on realistic arrival patterns, token lengths, prefill–decode balance, temporal locality, and prefix reuse.Yet evaluations often rely on ShareGPT-derived requests, chat, codecompletion, summarization traces, or short and partially released production traces.

9 Conclusion

The paper presents a one-year production trace of LLM serving and uses it to characterize workload evolution, user–model interactions, prefix caching, and load-balancing behavior. It finds that production workloads are dynamic and heterogeneous, with changing aggregate demand, diverse usage patterns, uneven cache locality, and coupled caching and load balancing.

  • Conclusion: The study presents a one-year production trace of LLM serving for characterizing workload evolution, user–model interactions, prefix caching, and load-balancing behavior.The trace is used as the basis for the paper’s workload analysis.
  • Conclusion: Production LLM workloads are dynamic and heterogeneous, with aggregate demand changing over time and users and models exhibiting diverse usage patterns.These patterns are identified through the paper’s workload characterization.
  • Conclusion: Cache locality is unevenly distributed across the workload, while prefix caching and load balancing are closely coupled.The conclusion identifies this coupling as a fundamental relationship in production serving behavior.
Loading 2608.13573v1…