Vidur: A Large-Scale Simulation Framework For LLM Inference

Amey Agrawal; Nitin Kedia; Jayashree Mohan; Ashish Panwar; Nipun Kwatra; Bhargav Gulavani; Ramachandran Ramjee; Alexey Tumanov

Source-linked AI summary

Vidur: A Large-Scale Simulation Framework For LLM Inference

Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee, Alexey Tumanov

arXiv:2405.05465v2May 8, 2024cs.LGcs.AIcs.CL

TL;DR

LLM deployment is costly to optimize because many configuration choices interact with model and workload characteristics. Vidur combines profiling, predictive runtime estimation, and simulation, while Vidur-Search automates deployment configuration search. Vidur predicts request-level performance with under 9% error and supports substantially cheaper exploration than deployment-based search.

Problem
LLM deployment optimization requires expensive experiments across large spaces of parallelization, scheduling, batching, hardware, and workload configurations.
Method
Vidur profiles a minimal set of model operators and uses predictive runtime estimation to simulate inference across configurations and workloads, with Vidur-Search exploring deployment choices.
Results
Under 9% error is achieved for request-level LLM inference performance, with high-fidelity aggregate metrics across large-scale workloads and traces.
Takeaways & Limitations
Vidur and its benchmark and search suite support deployment what-if analysis and identify efficient strategies at nominal cost.
Takeaways & Limitations
Vidur currently supports only synchronous pipeline-parallel scheduling and leaves asynchronous communication, sequence parallelism, and speculative pipelined decoding for future extension.

Abstract

from arXiv · show

Optimizing the deployment of Large language models (LLMs) is expensive today since it requires experimentally running an application workload against an LLM implementation while exploring large configuration space formed by system knobs such as parallelization strategies, batching techniques, and scheduling policies. To address this challenge, we present Vidur - a large-scale, high-fidelity, easily-extensible simulation framework for LLM inference performance. Vidur models the performance of LLM operators using a combination of experimental profiling and predictive modeling, and evaluates the end-to-end inference performance for different workloads by estimating several metrics of interest such as latency and throughput. We validate the fidelity of Vidur on several LLMs and show that it estimates inference latency with less than 9% error across the range. Further, we present Vidur-Search, a configuration search tool that helps optimize LLM deployment. Vidur-Search uses Vidur to automatically identify the most cost-effective deployment configuration that meets application performance constraints. For example, Vidur-Search finds the best deployment configuration for LLaMA2-70B in one hour on a CPU machine, in contrast to a deployment-based exploration which would require 42K GPU hours - costing ~218K dollars. Source code for Vidur is available at https://github.com/microsoft/vidur.

1 INTRODUCTION

LLM deployment optimization is expensive because performance depends on many configuration choices and on the specific model-workload pair. Vidur, Vidur-Bench, and Vidur-Search address this challenge through high-fidelity simulation, workload coverage, and automated configuration search.

Motivation: LLM deployment requires jointly selecting parallelization, scheduling, batching, and other parameters while testing representative workloads, making systematic optimization expensive and impractical.The configuration space spans model parallelism, scheduling algorithms, batch size, wait time, and algorithm-specific parameters.
Motivation: Up to 2× cost differential can result when a configuration optimized for one workload trace is applied to the same model on another trace.Optimal deployment depends on the model-trace pair, and new models and traces continually expand the search burden.
Vidur: Vidur combines operator decomposition, minimal experimental profiling, and runtime estimation to simulate LLM inference across deployment scenarios.It identifies operators and profiled input sizes, then predicts kernel performance for unprofiled sizes.
Results: Vidur achieves under 9% error for request-level inference performance and mimics aggregate cluster metrics for large-scale workloads and traces.The paper evaluates fidelity across a range of models, hardware, and cluster configurations.
Contributions: Vidur-Bench provides extensible workload traces together with batching and scheduling policies to address the lack of a standardized comprehensive LLM inference benchmark.The benchmark includes policies such as vLLM, Orca, FasterTransformer, and Sarathi-Serve.
Contributions: Vidur-Search identifies high-throughput-per-cost deployment configurations automatically, finding a LLaMA2-70B configuration in about one hour on a CPU machine.The alternative deployment-based exploration would require 42K GPU hours and approximately $218K.

2 BACKGROUND AND MOTIVATION

LLM inference efficiency involves trade-offs across parallelism, scheduling, hardware, and workload-sensitive configuration choices. Because the configuration space scales with both models and traces, the paper motivates simulation-based search as a lower-cost alternative to exhaustive deployment experiments.

Inference Basics: LLM inference uses prefill to process the prompt and decode to generate output tokens autoregressively.The decode phase repeatedly generates one token until the end-of-sequence token is produced.
Efficiency Optimizations: Tensor parallelism can improve throughput and latency but requires frequent communication over expensive high-bandwidth interconnects.It shards model weights and KV-Cache across GPU workers.
Efficiency Optimizations: Pipeline parallelism improves the compute-communication ratio relative to tensor parallelism but can suffer from pipeline bubbles caused by stage imbalance.The model is partitioned into consecutive transformer-block stages across GPUs.
Scheduling: Prefill-prioritizing schedulers favor throughput, whereas decode-prioritizing schedulers favor latency at the cost of lower throughput.The scheduler choice therefore exposes a cost-latency trade-off.
Configuration Space: Configuration search has complexity O(|M| · |T|) because optimal deployment depends on both the model set and workload-trace set.Using one trace’s optimal configuration on another can produce up to a 2× cost differential.
Configuration Search: Simulation-based search is proposed to find performant configurations without expensive experimental resources and reduce search cost by several orders of magnitude.The motivation follows from the high cost of obtaining individual deployment measurements and the expanding model-trace search space.

3 CHALLENGES IN SIMULATING LLM INFERENCE

LLM inference simulation is difficult because it requires fine-grained timing, variable iteration behavior, and resistance to cascading prediction errors. These challenges distinguish inference simulation from conventional DNN-training simulation.

Time Scale: LLM inference iterations can last only a few milliseconds, requiring substantially finer-grained timing predictions than conventional DNN training.Training iterations commonly run for hundreds of milliseconds.
Varying Iteration Times: Inference iteration latency varies because prefill and decode have different compute characteristics and request inputs can differ substantially.Dynamic request composition and interleaving of inference phases contribute to runtime variation.
Cascading Errors: Small errors in individual batch-runtime predictions can alter later batching decisions and cascade into aggregate simulation errors.Dynamic request arrivals make inference batches interdependent over time.
Simulator Architecture: Vidur’s architecture is presented as a high-level simulator design for addressing these inference-specific challenges.The simulator includes hierarchical scheduling and detailed request- and cluster-level metrics.

4 VIDUR

Vidur simulates LLM inference by combining operator-level profiling, predictive runtime estimation, and hierarchical scheduling. Its decomposition of operators and shared architectural structure reduces profiling needs while supporting detailed workload and deployment simulations.

System Overview: Vidur emulates model execution and request scheduling at replica and cluster levels to estimate LLM inference performance.The simulator supports both model execution and multiple tiers of scheduling.
System Overview: Vidur exploits shared LLM architectures to represent models declaratively and model a small set of reusable compute operators.Architectural similarities reduce the number of model-specific operators that must be represented.
Runtime Estimation: Vidur profiles minimal operator inputs and trains small predictive models to generate runtime lookup tables across unprofiled parameter ranges.The onboarding phase feeds profiled runtimes to the estimator, which produces operation-wise tables for simulation.
Operator Profiling: The profiler triages operators by whether runtime depends on total tokens, request context lengths, or communication characteristics.The three categories are token-level, sequence-level, and communication operators.
Sequence-Level Operators: Prefill attention is approximated separately from decode attention, while decode runtime is modeled primarily from total KV-Cache reads.Decode attention is treated as memory-bound, whereas prefill attention depends quadratically on prefill length.
Hierarchical Scheduler: Vidur uses a three-tier hierarchical scheduler supporting routing, batching, memory management, and multiple scheduling policies.The replica-stage scheduler currently supports synchronous pipeline-parallel scheduling, with additional optimizations planned.

5 VIDUR-BENCH

Vidur-Bench is an extensible benchmark suite for evaluating LLM inference across workloads, scheduling and batching policies, and serving frameworks. It provides curated traces and system-level metrics for performance analysis and tuning.

Benchmark Scope: Vidur-Bench supports plug-and-play workload patterns, scheduling, batching, routing policies, and serving frameworks.Its scope is designed for flexible evaluation of different inference-system configurations.
Workloads: Workload characteristics strongly affect inference performance, including input and output token counts and the relative costs of prefill and decode.The decode phase can be as high as 200× more expensive than prefill.
Workloads: Vidur-Bench curates workloads from public datasets for evaluating varying request types and arrival rates.These workloads can also be used to tune performance-sensitive serving-system parameters.
Performance Metrics: The suite provides system-level performance metrics for analyzing inference behavior.These include operator inputs and execution times, request scheduling delay, prefill completion time, TTFT, and TBT.

6 VIDUR-SEARCH

Vidur-Search searches deployment configurations under application constraints to identify cost-effective LLM serving choices. It evaluates configurations across hardware, parallelism, scheduling, batching, and workload settings using Vidur simulations.

Inputs and Constraints: Vidur-Search takes a model, workload, available GPU SKUs, replica GPU limits, and latency constraints as search inputs.The constraints include SLOs such as TTFT and TBT.
Search Space: The search space includes parallelism strategy and degree, scheduling policies, scheduler parameters, batch size, and GPU SKU.These knobs correspond to common deployment choices such as TP versus PP and scheduler-specific chunk sizes.
Optimization Objective: Vidur-Search maximizes QPS per dollar while constraining P99 scheduling delay below 5 seconds.Capacity is defined as the maximum supported QPS without unbounded queuing delay, divided by hourly GPU cost.
Search Procedure: The tool enumerates deployment configurations and uses simulation to predict metrics such as TTFT and TBT at selected QPS values.This converts configuration evaluation into a constrained optimization problem rather than repeated hardware experiments.
Search Motivation: Simulation reduces the cost of exploring thousands of configurations when optimal choices vary with workload and workloads change over time.The paper motivates repeating searches as workload characteristics evolve.
Scope: Vidur-Search can also optimize offline inference by replacing QPS per dollar with an objective such as makespan.Its primary design target is online serving, but the objective function can be changed for offline scenarios.

7 EVALUATION

Vidur is evaluated across diverse models, hardware, workloads, schedulers, and deployment configurations to test fidelity and support what-if analysis. It predicts request-level performance accurately while revealing that workload characteristics strongly affect optimal deployments and serving cost.

Evaluation Setup: Vidur is evaluated across four models, A100 and H100 hardware, and traces derived from three real-world workload datasets.The workloads capture different prompt and decode-length distributions, including Chat-1M, Arxiv-4K, and BWB-4K.
Simulator Fidelity: 3.33% maximum error: Vidur predicts static-workload P95 request latency across four models and three datasets.The evaluation attributes slightly higher average error for the 7B model to higher CPU overhead.
Simulator Fidelity: < 5% error: Vidur achieves high fidelity in almost all dynamic-workload scenarios at 85% of system capacity.Near capacity, small prediction differences can trigger sharp queue-delay increases, making the operating point especially sensitive.
What-if Analysis: $125: simulation-based exploration costs far less than the $1.14 million estimated for actual execution.Workload changes also alter the optimal batch size, GPU SKU, and deployment configuration; for LLaMA2-70B, Chat-1M uses batch size 256 and H100, whereas BWB uses 64 and A100.
What-if Analysis: 2× overhead: applying an LLaMA2-70B configuration optimized for Arxiv-Summarization-4K to LMSys-Chat-1M can substantially worsen cost or performance.The result demonstrates that optimal configurations depend on the model-trace pair.
What-if Analysis: A 20ms TBT SLO relaxation from 0.12 to 0.14 seconds changes the Pareto point from approximately 0.07 to 0.13, a ∼1.85× cost reduction.Configurations optimal for one latency metric may still violate the other metric’s SLO.

8 RELATED WORK

Vidur builds on training simulators that model predictable iterations, computation graphs, and parallelization strategies, but targets the distinct variability and configuration complexity of LLM inference. The related figures emphasize deployment trade-offs involving latency SLOs, Pareto efficiency, and QPS per dollar.

Prior Training Simulators: Training simulators model job performance using iteration profiles, roofline-based operator estimates, graph transformations, or distributed strategy representations.The cited approaches include Habitat, Daydream, and Proteus.
Deployment Trade-offs: Figure 5 compares capacity per dollar with TTFT-P90 and TBT-P99, marking SLO-satisfying regions and visualizing Pareto configurations.Green points satisfy both SLOs, while blue Pareto points can violate the other latency metric’s SLO.
Deployment Trade-offs: Figure 6 reports QPS per dollar for the best configurations under P90 TTFT and P99 TBT SLOs of 2s and 200ms.The comparison summarizes cost-normalized capacity under fixed latency constraints.

9 CONCLUSION

Vidur is presented as a high-fidelity, extensible simulator, benchmark, and search suite for evaluating and optimizing LLM inference deployments. It addresses the impracticality of testing the many deployment configurations that affect inference efficiency.

Conclusion: Vidur combines a high-fidelity, extensible LLM inference simulator with a benchmark and search suite.The tools answer deployment what-if questions and evaluate system optimizations at nominal cost.
Conclusion: The framework targets inference efficiency across configuration knobs including parallelism, scheduling strategy, and GPU SKU.These knobs make exhaustive evaluation on actual hardware impractical.

A.1 Impact of Request Arrival Rate on Fidelity for Dynamic Workloads

Vidur retains high fidelity as request arrival rates increase, including at 95% of maximum capacity for larger models. LLaMA2-7B is an exception, where CPU overhead causes error to cascade under high load.

Impact of Request Arrival Rate on Fidelity for Dynamic Workloads: 95% of maximum capacity: Vidur retains high fidelity for larger models at high request arrival rates.Additional arrival-rate experiments extend the fidelity evaluation beyond the main operating point.
Impact of Request Arrival Rate on Fidelity for Dynamic Workloads: 12.65% maximum error: LLaMA2-7B shows cascading errors at high arrival rates because of CPU overhead.The smaller model’s higher CPU overhead limits fidelity as load increases.

A.2 Cost Breakdown of What-if Analysis

Vidur enables what-if exploration of deployment configurations at far lower cost than actual execution. The analysis covers 35,565 runs and completes on a 96-core CPU machine in approximately 12.5 hours.

35,565 runs in the what-if analysis would require a projected GPU cost of $1,139,865 with actual execution.The same search completes in approximately 12.5 hours on a 96-core CPU machine.
Table 2 reports the cost of finding the optimal deployment configuration.