PaperlayerView exact revision on arXiv

Source-linked AI summary

Splitwise: Efficient generative LLM inference using phase splitting

Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini

arXiv:2311.18677v2cs.ARcs.DC

TL;DR

LLM inference efficiency is challenging because prompt computation is compute-intensive while token generation is memory-intensive and underuses compute resources. Splitwise separates these phases across machines, transfers cached state between them, and provisions phase-specific clusters. The resulting designs achieve higher throughput at lower cost, including 1.4× higher throughput at 20% lower cost than existing clusters and 2.35× more throughput under the same cost and power budgets.

  • Problem

    LLM inference uses expensive, power-hungry accelerators even though prompt computation and token generation have distinct resource characteristics.

  • Method

    Splitwise separates prompt computation and token generation onto different machines, transferring cached context over optimized datacenter interconnects and provisioning resources independently.

  • Results

    1.4× higher throughput at 20% lower cost is achieved than existing clusters, or 2.35× more throughput with the same cost and power budgets.

  • Takeaways & Limitations

    Splitwise supports cluster designs optimized for throughput, cost, and power across homogeneous and heterogeneous deployments using production traces.

  • Takeaways & Limitations

    The study evaluates NVIDIA H100 and A100 GPUs and lacks access to alternative hardware such as CPUs, FPGAs, and ASICs.

Abstract

from arXiv · show

Recent innovations in generative large language models (LLMs) have made their applications and use-cases ubiquitous. This has led to large-scale deployments of these models, using complex, expensive, and power-hungry AI accelerators, most commonly GPUs. These developments make LLM inference efficiency an important challenge. Based on our extensive characterization, we find that there are two main phases during an LLM inference request: a compute-intensive prompt computation, and a memory-intensive token generation, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Specifically, unlike compute-intensive prompt computation phases, token generation phases do not require the compute capability of the latest GPUs, and can be run with lower power and cost. With Splitwise, we propose splitting the two phases of a LLM inference request on to separate machines. This allows us to use hardware that is well-suited for each phase, and provision resources independently per phase. However, splitting an inference request across machines requires state transfer from the machine running prompt computation over to the machine generating tokens. We implement and optimize this state transfer using the fast back-plane interconnects available in today's GPU clusters. We use the Splitwise technique to design LLM inference clusters using the same or different types of machines for the prompt computation and token generation phases. Our clusters are optimized for three key objectives: throughput, cost, and power. In particular, we show that we can achieve 1.4x higher throughput at 20% lower cost than current designs. Alternatively, we can achieve 2.35x more throughput with the same cost and power budgets.

I. INTRODUCTION

Generative LLM inference has distinct compute-intensive prompt and memory-intensive token-generation phases, motivating Splitwise’s separation of these phases across machines. The resulting clusters target throughput, cost, and power through phase-specific hardware and resource management.

  • I. INTRODUCTION: Generative LLM inference comprises a parallel prompt computation phase and sequential token generation phase with contrasting resource demands.Prompt computation requires high GPU FLOPs, while token generation relies more on memory bandwidth and capacity.
  • I. INTRODUCTION: 3.43× more compute and 1.75× more power characterize H100 over A100, while memory bandwidth grew only 1.6× and capacity did not increase.The hardware scaling mismatch motivates matching different resources to the two inference phases.
  • I. INTRODUCTION: Splitwise separates prompt computation and token generation onto different machines to manage resources independently and use better-suited hardware.The design transfers the cached context from the prompt machine to the token-generation machine over datacenter interconnects.
  • I. INTRODUCTION: 1.4× higher throughput at 20% lower cost is achieved by Splitwise-based clusters than by existing clusters.Alternatively, the clusters deliver 2.35× more throughput under the same cost and power budgets.
  • I. INTRODUCTION: The paper characterizes phase differences, introduces Splitwise, explores homogeneous and heterogeneous deployments, and evaluates them using production traces.The stated objectives are overall cost, request throughput, and provisioned power.

C. Performance metrics for LLMs

LLM inference is evaluated through latency and throughput metrics shaped by workload requirements, batching mechanisms, and model-parallel execution. Mixed batching allows prompt and token phases to share execution while reducing, but not eliminating, token latency interference.

  • C. Performance metrics for LLMs: E2E latency, TTFT, and throughput are established inference metrics, while TBT measures streaming time between generated tokens.Different task types prioritize these metrics differently: batch tasks emphasize throughput, whereas conversational APIs emphasize TTFT and TBT.
  • D. Batching of requests: Request-level batching can cause long waits for arriving requests when existing requests have lengthy token-generation phases.This produces high TTFT and E2E latency.
  • D. Batching of requests: Mixed batching makes scheduling decisions at each forward pass so prompt and token phases can run together.It reduces TBT impact but does not eliminate the longer runtime experienced by token phases scheduled with prompt phases.
  • C. Performance metrics for LLMs: Pipeline parallelism divides model layers across GPUs, whereas tensor parallelism divides tensors across GPUs while replicating layers.Pipeline parallelism requires less communication; tensor parallelism requires high-bandwidth communication.
  • III. CHARACTERIZATION: The characterization uses production traces from Azure coding and conversation services to represent common LLM inference scenarios.The traces guide input and output token sizes rather than prompt content.

A. Number of prompt and generated tokens

Prompt and generated-token distributions vary substantially across services, while batching and latency measurements reveal contrasting behavior between prompt computation and token generation. Prompt work is compute-bound, whereas token generation dominates end-to-end time and benefits more from batching.

  • A. Number of prompt and generated tokens: 1500 median prompt tokens occur in coding traces, compared with 1020 in conversation traces.Coding prompts contain large code prefixes, while conversation prompts span a wider range.
  • A. Number of prompt and generated tokens: 13 median output tokens occur in coding traces, compared with 129 in conversation traces.The coding service generates only a few next words, while conversation outputs have an almost bimodal distribution.
  • A. Number of prompt and generated tokens: Different inference services may have widely different prompt and token distributions.This variation affects how systems should be characterized and provisioned.
  • B. Batch utilization: 60–70% of conversation time is spent with 20 active tokens or fewer in mixed continuous batching.Coding experiences even worse token-phase batching because it generates very few output tokens.
  • B. Batch utilization: Mixed continuous batching spends most of its time with very few active tokens batched.This indicates substantial underutilization during token generation.
  • C. Latency: Prompt TTFT grows almost linearly with prompt size because prompt computation is highly utilized and compute-bound.By contrast, batching output tokens has little TBT impact; batch size 64 produces only a 2× impact on TBT.
  • C. Latency: Most end-to-end latency is spent in token generation, including for coding requests with large prompts and few generated tokens.For BLOOM-176B, 1500 prompt tokens take the same time as only 6 generated tokens in the token phase.
  • D. Throughput: Prompt throughput decreases after 2048 prompt tokens, whereas token-phase throughput increases with batching until batch size 64, when memory is exhausted.The prompt phase therefore benefits from limiting batch size, while token-phase batching increases throughput without the reported downside.

E. Memory utilization

Splitwise characterizes prompt computation as compute- and power-intensive, while token generation is constrained by memory capacity and underuses GPU power. These differences motivate phase-specific hardware and power management.

  • Token generation accesses the KV cache of the entire context for every active generated token.The KV cache grows during generation, increasing memory-capacity requirements as batch tokens increase.
  • Prompt-phase batching is compute-bound, whereas token-phase batching is limited by memory capacity.Token throughput continues scaling with batch size until the machine runs out of memory.
  • Token generation incurs almost no latency impact when power is capped from 700 to 350W.Prompt latency is highly sensitive to power caps, unlike token-generation latency.
  • The token phase does not utilize the GPU power budget as efficiently as the prompt phase.This supports using less compute-capable hardware for token generation to improve performance per watt and performance per dollar.
  • Token generation can run on less compute-capable hardware for better Perf/W and Perf/$ efficiencies.A100 provides better or equal overall inference cost and energy compared with H100 in the reported comparison.

A. Cluster-level scheduling

Splitwise uses hierarchical scheduling to manage separate prompt, token, and mixed machine pools. Cluster-level routing assigns requests across pools, while machine-level schedulers batch according to compute and memory constraints.

  • The cluster-level scheduler manages prompt, token, and mixed pools and routes incoming inference requests.Machines can move into the mixed pool to reduce fragmentation and meet service-level objectives at higher loads.
  • Mixed-pool machines expand or contract with workload demand and return to their original pools after opposite-kind work drains.Switching pools does not incur noticeable latency.
  • Join the Shortest Queue assigns each request both a prompt machine and a token machine using pending-token queue lengths.Simultaneous assignment allows KV-cache transfers to overlap with prompt computation.
  • The machine-level scheduler restricts prompt batches to 2048 total prompt tokens because larger batches degrade throughput.Prompt scheduling uses FCFS, while the 2048-token limit is configurable by model or hardware.
  • Token machines batch as much as possible while queueing tokens when GPU memory approaches exhaustion.Token throughput scales with batch size until the machine runs out of memory.
  • Mixed machines prioritize prompts and may preempt tokens to meet TTFT SLOs.Token age priorities and bounded preemptions reduce starvation of token generation.

C. KV-cache transfer

KV-cache transfer is Splitwise’s main overhead because prompt-generated state must move between machines before token generation continues. Layer-wise asynchronous transfers overlap communication with computation, while provisioning searches balance latency, throughput, cost, and power.

  • KV-cache transfer is the main overhead introduced by splitting prompt computation and token generation across machines.The cache is generated during prompting and grows during token generation.
  • Serialized KV-cache transfer delays the next output token and directly increases maximum TBT and end-to-end latency.Transfer time depends on cache size and interconnect bandwidth, and can significantly affect TBT for large prompts.
  • Layer-wise asynchronous transfer sends each layer’s KV cache while the prompt machine computes the next layer.Fine-grained per-layer synchronization can interfere with computation, especially for smaller prompts.
  • Splitwise-HA uses DGX-H100 prompt machines and DGX-A100 token machines because A100s can be more cost- and power-efficient for token generation.Splitwise-HHcap instead power-caps H100 token machines based on their lower sensitivity to power reduction.
  • Provisioning searches use a simulator to select prompt and token machine counts satisfying latency SLOs while optimizing throughput, cost, or power.For Splitwise-HH coding workloads, 27 prompt and 3 token machines achieve 70 RPS at the lowest cost.

E. Practical Considerations

Splitwise preserves inference accuracy and is evaluated with real hardware, performance modeling, and an event-driven simulator. Practical boundaries include a possible cluster-level scheduling bottleneck and out-of-scope failure recovery.

  • Splitwise does not impact accuracy because KV-cache transfer is lossless and inference uses the same parameters and state as a single machine.
  • The cluster-level scheduler may become a scalability bottleneck for large clusters.Improving this scalability is identified as orthogonal to Splitwise.
  • Failure recovery is out of scope; machine failures restart requests from scratch under the current design.Checkpointing KV caches is presented as an alternative recovery approach rather than an implemented mechanism.
  • The real-hardware implementation uses modified vLLM on DGX-A100 and DGX-H100 virtual machines connected by InfiniBand.DGX-H100 connections provide double the bandwidth, at 400 Gbps.
  • The event-driven simulator models machine pools, schedulers, memory, queues, and KV-cache transfer for large-scale evaluation.It uses production request distributions and reports TTFT, TBT, E2E latency, and machine utilization.
  • The performance model has a mean absolute percentage error below 3% on an 80:20 train:test split.
  • Evaluation enforces P50, P90, and P99 SLOs for TTFT, TBT, and E2E latency, requiring all nine SLOs to be met.

VI. EVALUATION

The evaluation measures KV-cache transfer overhead and compares Splitwise cluster designs under iso-power workloads. Splitwise improves latency, batching, and throughput, with workload-dependent trade-offs across cluster configurations.

  • KV-cache transfer: < 7% KV-cache transfer overhead is observed relative to prompt computation on A100 and H100 setups.Optimized per-layer transfer maintains about 8 ms non-overlapped time on A100 and 5 ms on H100.
  • KV-cache transfer: 2× faster transfers occur on H100 than A100 because the H100 setup provides 400 Gbps versus 200 Gbps bandwidth.
  • End-to-end impact: 0.8% E2E latency overhead is incurred by Splitwise, versus up to 3% for serialized transfer with large prompts.Splitwise adds 16.5% latency to the second token, compared with 64% for serialized transfer.
  • Cluster latency: Splitwise-HHcap performs best across latency metrics for the conversation trace because its token phases typically run longer.
  • Batched tokens: 70% of baseline conversation-cluster time is spent running 15 or fewer active tokens, while Splitwise machines batch more effectively at low load.At high load, batch sizes become more similar across prompt and token machines.
  • Cluster throughput: 2.15× more throughput is achieved by Splitwise-AA than Baseline-A100 at the same power and cost.Splitwise-HA achieves 1.18× more throughput at 10% lower cost and the same power.

C. Other cluster optimizations

Additional cluster optimizations show that Splitwise can improve throughput, cost, and power, while its behavior remains robust across workload and model changes.

  • Iso-cost throughput optimization: 1.4× more throughput is achieved by Splitwise-AA than Baseline-H100 at the same cost, with 25% more power and 2× the space.
  • Iso-throughput power optimization: 25% lower power is achieved by Splitwise-HHcap at the same throughput, cost, and space as Baseline-H100.
  • Iso-throughput cost optimization: 25% lower cost is achieved by Splitwise-AA at the same throughput as Baseline-H100.
  • Workload robustness: 7% throughput setbacks occur for Splitwise-HA and Splitwise-HHcap when workload changes because their prompt and token pools use different machine types.Splitwise-AA and Splitwise-HH with mixed pools show no throughput or latency impact in the workload-change experiment.
  • Model robustness: All Splitwise designs outperform both baselines at higher load when serving Llama-70B on a cluster designed for BLOOM-176B.Splitwise-HH and Splitwise-HHcap consistently achieve the best latency as load increases.
  • Robustness summary: Splitwise adapts to changes in workload, LLM, request load, and token distributions through smart scheduling.

E. Cluster design for batch job

For batch-oriented settings and broader deployment, the paper discusses high-load behavior, applicability across transformer-based LLMs, and hardware and interconnect boundaries.

  • Batch-job design: 0.89 RPS/$ is the best throughput-per-cost achieved by Baseline-A100 and Splitwise-AA under stressed iso-power clusters.At high load, Splitwise devolves into the iso-count baseline as all machines begin mixed batching.
  • Extensibility: Splitwise applies to current and upcoming autoregressive transformer-based LLMs, including Mixture-of-Experts models.The technique relies on the presence of distinct prompt-processing and token-generation phases.
  • Extensibility: Clusters provisioned with Splitwise for one model can also efficiently serve other models.
  • Alternative hardware: Prompt phases need high compute capability and memory bandwidth with low memory capacity, whereas token phases need moderate compute capability with high memory capacity and bandwidth.The paper leaves evaluation of alternative hardware such as AMD MI-250 and Intel Sapphire Rapids for future work.
  • Interconnect boundary: Splitwise assumes InfiniBand connections between prompt and token machines, and heterogeneous Splitwise-HA may not be readily available with such connectivity.
  • Related systems: Existing model-serving approaches generally use the same machine for prompt and token phases, whereas Splitwise separates them.

APPENDIX

The released artifact provides traces, a KV-cache transfer prototype, and simulation components for evaluating Splitwise and future LLM serving systems.

  • Artifact contents: Production traces from two Microsoft Azure LLM inference services are included in the artifact.
  • Artifact contents: A prototype implementation of Splitwise’s KV-cache transfer mechanism in vLLM is included.
  • Artifact contents: SplitwiseSim is a discrete-event simulator for evaluating model serving in LLM inference clusters.Artifact functionality was tested only for the traces and SplitwiseSim because of limited hardware availability.
  • Runtime and hardware: The vLLM prototype runs on two machines connected over GPU InfiniBand, while SplitwiseSim runs on an x86-64 CPU machine.
  • Availability: The artifact is publicly available under MIT code and CC-BY data licenses.
  • Availability: The artifact is archived under DOI 10.5281/zenodo.11003049.

C. Description

The artifact is distributed through Zenodo and several online repositories, with components covering traces, KV-cache transfer, and SplitwiseSim. It documents hardware and software dependencies for the transfer prototype and simulation environment.

  • Artifact Availability: The complete artifact is available as a Zenodo archive, with individual components distributed through separate online repositories.The listed components include production traces, a KV-cache transfer prototype, and SplitwiseSim with experiment and plotting scripts.
  • Hardware Dependencies: The KV-cache transfer prototype requires two GPU machines connected over Infiniband.Examples include NVIDIA DGX-A100 or DGX-H100 systems.
  • Hardware Dependencies: SplitwiseSim runs on a standard x86-64 CPU machine, with multiple machines available to parallelize simulation runs.
  • Software Dependencies: The transfer prototype builds on vLLM and MSCCL++, while SplitwiseSim uses publicly available Python packages installed through requirements.txt.
  • Data Sets: Coding and conversation traces from Microsoft Azure are included online as part of the artifact release.
  • Workflow: Installation and usage instructions are provided in the artifact README files.
Loading 2311.18677v2…