Source-linked AI summary
BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems
Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, Xiaowen Chu
TL;DR
Open-source LLM serving workloads are scarce, so serving optimizations are often evaluated with unrealistic assumptions that may not generalize to real-world LLM workloads. BurstGPT collects 10.10 million regional Azure OpenAI GPT traces over 213 days, covering request timing, token lengths, service types, LLM types, and failures, and provides lightweight, modular evaluation tooling. BurstGPT evaluations reveal declines in serving efficiency, stability, or reliability under realistic workload variation, while optimization generalization across serving types is not guaranteed.
Problem
Open-source LLM serving workloads are scarce, so serving optimizations are often evaluated with unrealistic assumptions that may not generalize to real-world LLM workloads.
Method
BurstGPT collects 10.10 million regional Azure OpenAI GPT traces over 213 days, covering request timing, token lengths, service types, LLM types, and failures, and provides lightweight, modular evaluation tooling.
Results
BurstGPT evaluations reveal declines in serving efficiency, stability, or reliability under realistic workload variation, while optimization generalization across serving types is not guaranteed.
Takeaways & Limitations
BurstGPT supports workload-aware evaluation, provisioning, and optimization of LLM serving systems using realistic user, model, and system behavior.
Takeaways & Limitations
Serving optimizations have yet to be evaluated comprehensively on real-world LLM workloads, and generalization from synthetic or non-LLM workloads remains challenging.
Abstract
from arXiv · showhide
Serving systems for Large Language Models (LLMs) are often optimized to improve quality of service (QoS) and throughput. However, due to the lack of open-source LLM serving workloads, these systems are frequently evaluated under unrealistic workload assumptions. Consequently, performance may degrade when systems are deployed in real-world scenarios. This work presents BurstGPT, an LLM serving workload with 10.31 million traces from regional Azure OpenAI GPT services over 213 days. BurstGPT captures LLM serving characteristics from user, model and system perspectives: (1) User request concurrency: burstiness variations of requests in Azure OpenAI GPT services, revealing diversified concurrency patterns in different services and model types. (2) User conversation patterns: counts and intervals within conversations for service optimizations. (3) Model response lengths: auto-regressive serving processes of GPT models, showing statistical relations between requests and their responses. (4) System response failures: failures of conversation and API services, showing intensive resource needs and limited availability of LLM services in Azure. The details of the characteristics can serve multiple purposes in LLM serving optimizations, such as system evaluation and trace provisioning. In our demo evaluation with BurstGPT, frequent variations in BurstGPT reveal declines in efficiency, stability, or reliability in realistic LLM serving. We identify that the generalization of KV cache management, scheduling and disaggregation optimizations can be improved under realistic workload evaluations. BurstGPT is publicly available now at https://github.com/HPMLL/BurstGPT and is widely used to develop prototypes of LLM serving frameworks in the industry.
1 Introduction
BurstGPT addresses the mismatch between unrealistic evaluation workloads and real-world LLM serving by providing Azure OpenAI traces and methods for workload-aware optimization. Its analyses show that concurrency, conversation behavior, response lengths, and failures materially affect serving-system evaluation.
- Unrealistic synthetic and non-LLM workloads do not reflect auto-regressive LLM serving patterns, limiting the effectiveness of serving-system evaluation.
- Users can sample and scale BurstGPT or model its traces to reproduce realistic concurrency and response patterns for serving-system optimization and evaluation.Applications include provisioning concurrency patterns and predicting response lengths from request-response distributions.
- Non-LLM concurrency does not accurately reflect system performance, while optimization generalization across LLM serving types is not guaranteed.Optimized first-come-first-served scheduling is effective for conversation services but can be less efficient for API services.
- Longer and unpredictable responses increase serving workload and system-pressure uncertainty, while burstiness variations can create memory bottlenecks and failure-rate spikes.The analysis connects response-length distributions to response provisioning and identifies KV-cache management inefficiencies as a primary source of failures in the evaluation.
2 Preliminary and Motivation
Existing LLM serving optimizations are commonly evaluated with synthetic or non-LLM workloads, despite LLM-specific user, model, and system behaviors. BurstGPT motivates workload evaluation grounded in empirical concurrency and resource-use patterns.
- Prior serving systems optimize throughput, QoS, scheduling, memory, and compute, but their workload assumptions may not capture LLM-specific resource demands.
- Serving optimizations have not been adequately evaluated on real-world LLM workloads, making generalization from synthetic or non-LLM workloads challenging.
- LLM workloads vary with user, system, and model behavior, so neglecting any workload-trace aspect can yield an incomplete view of framework behavior in deployment.
- Synthetic and non-LLM concurrency lacks empirical LLM behavior; MAF averages 1.64 RPS versus 0.019 RPS in conversation and 0.21 RPS in ChatGPT API services.
- LLM prefilling and decoding scale as O(s^2) and O(s) with request length, creating resource occupation unlike lightweight cloud functions and synthetic workloads.
3 Introduction to BurstGPT
BurstGPT reveals distinct temporal, conversational, model-response, and failure patterns in real-world GPT services. These patterns vary across service types and models, challenging evaluations based on synthetic or non-LLM workloads.
- 3.1 User Request Concurrency: Conversation services show weekly and daily periodicity, with higher volumes on weekdays and during working hours, whereas API services exhibit irregular, bursty submissions.GPT-4 conversation usage is lower overall than ChatGPT, while API request-volume variance suggests possible automation.
- 3.1 User Request Concurrency: Burstiness varies between conversation and API services because their Gamma-distribution parameters differ over twenty-minute intervals.A smaller shape parameter α indicates a higher coefficient of variation and greater burstiness; conversation-service α changes sharply during working hours.
- 3.2 User Conversation Patterns: Over 35% of conversations contain one request, the median is two, and 75% contain four or fewer requests.Average interval time is not directly related to conversation length, while P90 intervals rise below five requests and then fluctuate.
- 3.2 User Conversation Patterns: Long intervals increase as the number of requests in a conversation increases across three duration thresholds and the IQR upper bound.These long intervals are defined as intervals exceeding a selected duration c.
4 BurstGPT-Perf: A Benchmark Suite
BurstGPT-Perf is a modular benchmark suite that reproduces BurstGPT’s streaming, stochastic, and bursty workloads at configurable scales. It evaluates serving systems using efficiency, stability, and reliability metrics, including latency jitters and failure rates.
- 4 BurstGPT-Perf: A Benchmark Suite: BurstGPT-Perf provides a lightweight, modular benchmark suite for evaluating LLM serving systems under streaming, stochastic, and bursty BurstGPT workloads.It also serves as an example implementation for deploying BurstGPT-based evaluations.
- 4.1 Workload Generator: The workload generator combines a Prompt Sampler, Prompt Pool, Concurrency Generator, and asynchronous HTTP client to produce evaluation requests.The Prompt Pool indexes prompts by token count to reduce sampling overhead.
- 4.2 Evaluation Workflow: The evaluation workflow samples prompts of specified lengths at stochastic intervals, treats concurrency as a time series, and sends requests to an inference engine over HTTP.Vanilla traces adjust RPS, while modeled traces adjust λ, α, and β according to a predetermined sequence.
- 4.3 Scaling Methods: BurstGPT supports RPS Scaling by multiplying timestamps and Modeled Scaling by adjusting concurrency-distribution parameters to match different serving-system sizes.Modeled scaling adjusts parameters at twenty-minute intervals; β can set a warm-up arrival rate compatible with system capacity.
- 4.3 Workload Modeling: Request lengths are sampled according to a Zipf distribution, while response lengths remain unconstrained so the model determines them dynamically.This preserves model-specific runtime behavior and lets output-length distributions emerge across requests.
- 4.4 Metrics and Setups: BurstGPT-Perf evaluates latency, throughput, latency jitters, and failure rates to measure efficiency, stability, and reliability.Metrics are reported as average or instantaneous values, with failure rate, token latency, latency-jitter standard deviation, and system throughput included.
5 Demo Evaluations
BurstGPT exposes workload-dependent changes in vLLM efficiency, reliability, and scheduling effectiveness. Its modeled traces also show that load-prediction accuracy depends on forecast granularity.
- 5.1 Evaluation: Real-world Workload: BurstGPT’s higher request-concurrency burstiness than MAF2 reveals workload-dependent serving behavior under equal RPS.The evaluation uses Llama-2-13b-chat on an A800 GPU server with both workloads.
- 5.2 Evaluation: Micro Workload Variations: Minor changes in α or CV can trigger sudden request failures without proportionate changes in average latency or throughput.Temporary quadratic changes affect reliability less visibly in aggregate performance metrics, whereas linear declines produce ongoing degradation during failures.
- 5.3 Evaluation: Workload-aware Scheduling: Scheduling effectiveness shifts across workloads: LRF improves latency and jitter over FCFS for conversation service but not API service.The comparison evaluates FCFS, SRF, and LRF using modeled BurstGPT conversation and API workloads.
- 5.4 Demo Use: Workload Provisioning: Load prediction uses request counts and mean tokens per request with short- and long-term historical features, plus calendar features, in XGBoost.The target is the actual load at each time point, evaluated with NMAE and NMSE.
- 5.4 Demo Use: Workload Provisioning: 58% lower NMSE is achieved for 10-minute prediction intervals than 1-minute intervals, making forecast granularity a scheduling trade-off.The authors report that long-term patterns are generally easier to predict than short-term patterns.
6 BurstGPT in Industry
BurstGPT supports industrial PD-disaggregation planning by simulating workload-dependent capacity and resource-ratio choices. Dynamic adjustment of the prefilling-to-decoding ratio improves goodput when workloads diverge.
- 6 BurstGPT in Industry: PD-disaggregation capacity depends on request concurrency, request lengths, and response lengths because prefilling and decoding are split across provider and consumer instances.BurstGPT supplies workload traces for selecting ratios before real-world deployment and for developing Huawei prototypes.
- 6 BurstGPT in Industry: BurstGPT simulations determine PD ratios by increasing QPS until the delay-SLO achievement rate is no longer met, defining system capacity.The procedure fixes total prefilling and decoding instances while testing candidate ratios against workload conditions.
- 6 BurstGPT in Industry: At 6:00, switching the PD ratio from 2:6 to 6:2 improves serving goodput under workload divergence.The comparison uses a fixed total number of instances and evaluates different static and dynamic ratios.
- 6 BurstGPT in Industry: The industry example demonstrates data-driven modeling of user and model behavior as a route to improving LLM-serving efficiency and QoS.The paper connects the simulation findings to careful scheduling-algorithm design.
7 Broader Impacts and Conclusions
The paper argues that realistic workload data is necessary for evaluating and optimizing LLM serving systems. BurstGPT’s traces expose workload-aware challenges and support improvements in efficiency, stability, and reliability.
- 7 Broader Impacts and Conclusions: BurstGPT addresses the shortage of real-world LLM workload data by revealing possible serving degradation under realistic workloads.The conclusion frames workload-aware optimization as both a challenge and an opportunity.
- 7 Broader Impacts and Conclusions: The authors encourage using BurstGPT to optimize and evaluate serving systems for efficiency, stability, and reliability under realistic workloads.They also advocate data-driven methodologies for future LLM-system development.