PaperlayerView exact revision on arXiv

Source-linked AI summary

Splitwise: Efficient generative LLM inference using phase splitting

Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini

arXiv:2311.18677v2cs.ARcs.DC

TL;DR

LLM inference efficiency is challenging because prompt computation is compute-intensive while token generation is memory-intensive and underuses compute resources. Splitwise separates these phases across machines, transfers cached state between them, and provisions phase-specific clusters. The resulting designs achieve higher throughput at lower cost, including 1.4× higher throughput at 20% lower cost than existing clusters and 2.35× more throughput under the same cost and power budgets.

  • Problem

    LLM inference uses expensive, power-hungry accelerators even though prompt computation and token generation have distinct resource characteristics.

    Source: §I · p. 1

  • Method

    Splitwise separates prompt computation and token generation onto different machines, transferring cached context over optimized datacenter interconnects and provisioning resources independently.

    Source: §I · p. 1

  • Results

    1.4× higher throughput at 20% lower cost is achieved than existing clusters, or 2.35× more throughput with the same cost and power budgets.

    Source: §I · p. 2

  • Takeaways & Limitations

    Splitwise supports cluster designs optimized for throughput, cost, and power across homogeneous and heterogeneous deployments using production traces.

    Source: §I · p. 2, §IX · p. 12

    The study evaluates NVIDIA H100 and A100 GPUs and lacks access to alternative hardware such as CPUs, FPGAs, and ASICs.

    Source: §VII · p. 12

Abstract

from arXiv · show

Recent innovations in generative large language models (LLMs) have made their applications and use-cases ubiquitous. This has led to large-scale deployments of these models, using complex, expensive, and power-hungry AI accelerators, most commonly GPUs. These developments make LLM inference efficiency an important challenge. Based on our extensive characterization, we find that there are two main phases during an LLM inference request: a compute-intensive prompt computation, and a memory-intensive token generation, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Specifically, unlike compute-intensive prompt computation phases, token generation phases do not require the compute capability of the latest GPUs, and can be run with lower power and cost. With Splitwise, we propose splitting the two phases of a LLM inference request on to separate machines. This allows us to use hardware that is well-suited for each phase, and provision resources independently per phase. However, splitting an inference request across machines requires state transfer from the machine running prompt computation over to the machine generating tokens. We implement and optimize this state transfer using the fast back-plane interconnects available in today's GPU clusters. We use the Splitwise technique to design LLM inference clusters using the same or different types of machines for the prompt computation and token generation phases. Our clusters are optimized for three key objectives: throughput, cost, and power. In particular, we show that we can achieve 1.4x higher throughput at 20% lower cost than current designs. Alternatively, we can achieve 2.35x more throughput with the same cost and power budgets.

I. INTRODUCTION

Generative LLM inference has distinct compute-intensive prompt and memory-intensive token-generation phases, motivating Splitwise’s separation of these phases across machines. The resulting clusters target throughput, cost, and power through phase-specific hardware and resource management.

I. INTRODUCTION

  • Generative LLM inference comprises a parallel prompt computation phase and sequential token generation phase with contrasting resource demands.Prompt computation requires high GPU FLOPs, while token generation relies more on memory bandwidth and capacity.Source: §I · p. 1, B. Generative LLM inference phases · p. 2
  • 3.43× more compute and 1.75× more power characterize H100 over A100, while memory bandwidth grew only 1.6× and capacity did not increase.The hardware scaling mismatch motivates matching different resources to the two inference phases.Source: §I · p. 1
  • Splitwise separates prompt computation and token generation onto different machines to manage resources independently and use better-suited hardware.The design transfers the cached context from the prompt machine to the token-generation machine over datacenter interconnects.Source: §I · pp. 1–2
  • 1.4× higher throughput at 20% lower cost is achieved by Splitwise-based clusters than by existing clusters.Alternatively, the clusters deliver 2.35× more throughput under the same cost and power budgets.Source: §I · p. 2
  • The paper characterizes phase differences, introduces Splitwise, explores homogeneous and heterogeneous deployments, and evaluates them using production traces.The stated objectives are overall cost, request throughput, and provisioned power.Source: §I · p. 2

C. Performance metrics for LLMs

LLM inference is evaluated through latency and throughput metrics shaped by workload requirements, batching mechanisms, and model-parallel execution. Mixed batching allows prompt and token phases to share execution while reducing, but not eliminating, token latency interference.

C. Performance metrics for LLMs

  • E2E latency, TTFT, and throughput are established inference metrics, while TBT measures streaming time between generated tokens.Different task types prioritize these metrics differently: batch tasks emphasize throughput, whereas conversational APIs emphasize TTFT and TBT.Source: C. Performance metrics for LLMs · p. 2

D. Batching of requests

  • Request-level batching can cause long waits for arriving requests when existing requests have lengthy token-generation phases.This produces high TTFT and E2E latency.Source: D. Batching of requests · p. 2
  • Mixed batching makes scheduling decisions at each forward pass so prompt and token phases can run together.It reduces TBT impact but does not eliminate the longer runtime experienced by token phases scheduled with prompt phases.Source: D. Batching of requests · p. 3

C. Performance metrics for LLMs

  • Pipeline parallelism divides model layers across GPUs, whereas tensor parallelism divides tensors across GPUs while replicating layers.Pipeline parallelism requires less communication; tensor parallelism requires high-bandwidth communication.Source: E. Model parallelism · p. 3

III. CHARACTERIZATION

  • The characterization uses production traces from Azure coding and conversation services to represent common LLM inference scenarios.The traces guide input and output token sizes rather than prompt content.Source: §III · p. 3

A. Number of prompt and generated tokens

Prompt and generated-token distributions vary substantially across services, while batching and latency measurements reveal contrasting behavior between prompt computation and token generation. Prompt work is compute-bound, whereas token generation dominates end-to-end time and benefits more from batching.

A. Number of prompt and generated tokens

B. Batch utilization

  • 60–70% of conversation time is spent with 20 active tokens or fewer in mixed continuous batching.Coding experiences even worse token-phase batching because it generates very few output tokens.Source: B. Batch utilization · p. 4
  • Mixed continuous batching spends most of its time with very few active tokens batched.This indicates substantial underutilization during token generation.Source: B. Batch utilization · p. 4

C. Latency

  • Prompt TTFT grows almost linearly with prompt size because prompt computation is highly utilized and compute-bound.By contrast, batching output tokens has little TBT impact; batch size 64 produces only a 2× impact on TBT.Source: C. Latency · p. 4
  • Most end-to-end latency is spent in token generation, including for coding requests with large prompts and few generated tokens.For BLOOM-176B, 1500 prompt tokens take the same time as only 6 generated tokens in the token phase.Source: C. Latency · p. 4

D. Throughput

  • Prompt throughput decreases after 2048 prompt tokens, whereas token-phase throughput increases with batching until batch size 64, when memory is exhausted.The prompt phase therefore benefits from limiting batch size, while token-phase batching increases throughput without the reported downside.Source: D. Throughput · p. 4

E. Memory utilization

Splitwise characterizes prompt computation as compute- and power-intensive, while token generation is constrained by memory capacity and underuses GPU power. These differences motivate phase-specific hardware and power management.

  • Token generation accesses the KV cache of the entire context for every active generated token.The KV cache grows during generation, increasing memory-capacity requirements as batch tokens increase.Source: E. Memory utilization · pp. 4–5
  • Prompt-phase batching is compute-bound, whereas token-phase batching is limited by memory capacity.Token throughput continues scaling with batch size until the machine runs out of memory.Source: E. Memory utilization · p. 5
  • Token generation incurs almost no latency impact when power is capped from 700 to 350W.Prompt latency is highly sensitive to power caps, unlike token-generation latency.Source: F. Power utilization · p. 5
  • The token phase does not utilize the GPU power budget as efficiently as the prompt phase.This supports using less compute-capable hardware for token generation to improve performance per watt and performance per dollar.Source: F. Power utilization · p. 5, G. GPU hardware variations · p. 5
  • Token generation can run on less compute-capable hardware for better Perf/W and Perf/$ efficiencies.A100 provides better or equal overall inference cost and energy compared with H100 in the reported comparison.Source: G. GPU hardware variations · p. 5

A. Cluster-level scheduling

Splitwise uses hierarchical scheduling to manage separate prompt, token, and mixed machine pools. Cluster-level routing assigns requests across pools, while machine-level schedulers batch according to compute and memory constraints.

  • The cluster-level scheduler manages prompt, token, and mixed pools and routes incoming inference requests.Machines can move into the mixed pool to reduce fragmentation and meet service-level objectives at higher loads.Source: A. Cluster-level scheduling · pp. 5–6
  • Mixed-pool machines expand or contract with workload demand and return to their original pools after opposite-kind work drains.Switching pools does not incur noticeable latency.Source: A. Cluster-level scheduling · p. 6
  • Join the Shortest Queue assigns each request both a prompt machine and a token machine using pending-token queue lengths.Simultaneous assignment allows KV-cache transfers to overlap with prompt computation.Source: A. Cluster-level scheduling · p. 6
  • The machine-level scheduler restricts prompt batches to 2048 total prompt tokens because larger batches degrade throughput.Prompt scheduling uses FCFS, while the 2048-token limit is configurable by model or hardware.Source: B. Machine-level scheduling · p. 6
  • Token machines batch as much as possible while queueing tokens when GPU memory approaches exhaustion.Token throughput scales with batch size until the machine runs out of memory.Source: B. Machine-level scheduling · p. 6
  • Mixed machines prioritize prompts and may preempt tokens to meet TTFT SLOs.Token age priorities and bounded preemptions reduce starvation of token generation.Source: B. Machine-level scheduling · p. 6

C. KV-cache transfer

KV-cache transfer is Splitwise’s main overhead because prompt-generated state must move between machines before token generation continues. Layer-wise asynchronous transfers overlap communication with computation, while provisioning searches balance latency, throughput, cost, and power.

  • KV-cache transfer is the main overhead introduced by splitting prompt computation and token generation across machines.The cache is generated during prompting and grows during token generation.Source: C. KV-cache transfer · p. 6
  • Serialized KV-cache transfer delays the next output token and directly increases maximum TBT and end-to-end latency.Transfer time depends on cache size and interconnect bandwidth, and can significantly affect TBT for large prompts.Source: C. KV-cache transfer · pp. 6–7
  • Layer-wise asynchronous transfer sends each layer’s KV cache while the prompt machine computes the next layer.Fine-grained per-layer synchronization can interfere with computation, especially for smaller prompts.Source: C. KV-cache transfer · p. 7
  • Splitwise-HA uses DGX-H100 prompt machines and DGX-A100 token machines because A100s can be more cost- and power-efficient for token generation.Splitwise-HHcap instead power-caps H100 token machines based on their lower sensitivity to power reduction.Source: D. Provisioning with Splitwise · p. 7
  • Provisioning searches use a simulator to select prompt and token machine counts satisfying latency SLOs while optimizing throughput, cost, or power.For Splitwise-HH coding workloads, 27 prompt and 3 token machines achieve 70 RPS at the lowest cost.Source: D. Provisioning with Splitwise · p. 7, Figure 12 · p. 7

E. Practical Considerations

Splitwise preserves inference accuracy and is evaluated with real hardware, performance modeling, and an event-driven simulator. Practical boundaries include a possible cluster-level scheduling bottleneck and out-of-scope failure recovery.

  • Splitwise does not impact accuracy because KV-cache transfer is lossless and inference uses the same parameters and state as a single machine.Source: E. Practical Considerations · p. 8
  • The cluster-level scheduler may become a scalability bottleneck for large clusters.Improving this scalability is identified as orthogonal to Splitwise.Source: E. Practical Considerations · p. 8
  • Failure recovery is out of scope; machine failures restart requests from scratch under the current design.Checkpointing KV caches is presented as an alternative recovery approach rather than an implemented mechanism.Source: E. Practical Considerations · p. 8
  • The real-hardware implementation uses modified vLLM on DGX-A100 and DGX-H100 virtual machines connected by InfiniBand.DGX-H100 connections provide double the bandwidth, at 400 Gbps.Source: A. Experimental setup · p. 8
  • The event-driven simulator models machine pools, schedulers, memory, queues, and KV-cache transfer for large-scale evaluation.It uses production request distributions and reports TTFT, TBT, E2E latency, and machine utilization.Source: B. Simulator setup · pp. 8–9
  • The performance model has a mean absolute percentage error below 3% on an 80:20 train:test split.Source: B. Simulator setup · p. 9
  • Evaluation enforces P50, P90, and P99 SLOs for TTFT, TBT, and E2E latency, requiring all nine SLOs to be met.Source: B. Simulator setup · p. 9

VI. EVALUATION

The evaluation measures KV-cache transfer overhead and compares Splitwise cluster designs under iso-power workloads. Splitwise improves latency, batching, and throughput, with workload-dependent trade-offs across cluster configurations.

KV-cache transfer

  • < 7% KV-cache transfer overhead is observed relative to prompt computation on A100 and H100 setups.Optimized per-layer transfer maintains about 8 ms non-overlapped time on A100 and 5 ms on H100.Source: A. Experimental results · p. 9
  • 2× faster transfers occur on H100 than A100 because the H100 setup provides 400 Gbps versus 200 Gbps bandwidth.Source: A. Experimental results · p. 9

End-to-end impact

  • 0.8% E2E latency overhead is incurred by Splitwise, versus up to 3% for serialized transfer with large prompts.Splitwise adds 16.5% latency to the second token, compared with 64% for serialized transfer.Source: A. Experimental results · p. 9

Cluster latency

Batched tokens

  • 70% of baseline conversation-cluster time is spent running 15 or fewer active tokens, while Splitwise machines batch more effectively at low load.At high load, batch sizes become more similar across prompt and token machines.Source: B. Iso-power throughput-optimized… · p. 10

Cluster throughput

  • 2.15× more throughput is achieved by Splitwise-AA than Baseline-A100 at the same power and cost.Splitwise-HA achieves 1.18× more throughput at 10% lower cost and the same power.Source: B. Iso-power throughput-optimized… · p. 10

C. Other cluster optimizations

Additional cluster optimizations show that Splitwise can improve throughput, cost, and power, while its behavior remains robust across workload and model changes.

Iso-cost throughput optimization

Iso-throughput power optimization

Iso-throughput cost optimization

Workload robustness

  • 7% throughput setbacks occur for Splitwise-HA and Splitwise-HHcap when workload changes because their prompt and token pools use different machine types.Splitwise-AA and Splitwise-HH with mixed pools show no throughput or latency impact in the workload-change experiment.Source: D. Impact of workload changes · p. 11

Model robustness

  • All Splitwise designs outperform both baselines at higher load when serving Llama-70B on a cluster designed for BLOOM-176B.Splitwise-HH and Splitwise-HHcap consistently achieve the best latency as load increases.Source: D. Impact of workload changes · p. 11

Robustness summary

E. Cluster design for batch job

For batch-oriented settings and broader deployment, the paper discusses high-load behavior, applicability across transformer-based LLMs, and hardware and interconnect boundaries.

Batch-job design

  • 0.89 RPS/$ is the best throughput-per-cost achieved by Baseline-A100 and Splitwise-AA under stressed iso-power clusters.At high load, Splitwise devolves into the iso-count baseline as all machines begin mixed batching.Source: E. Cluster design for batch job · p. 11

Extensibility

  • Splitwise applies to current and upcoming autoregressive transformer-based LLMs, including Mixture-of-Experts models.The technique relies on the presence of distinct prompt-processing and token-generation phases.Source: §VII · pp. 11–12
  • Clusters provisioned with Splitwise for one model can also efficiently serve other models.Source: §VII · p. 12

Alternative hardware

  • Prompt phases need high compute capability and memory bandwidth with low memory capacity, whereas token phases need moderate compute capability with high memory capacity and bandwidth.The paper leaves evaluation of alternative hardware such as AMD MI-250 and Intel Sapphire Rapids for future work.Source: §VII · p. 12

Interconnect boundary

  • Splitwise assumes InfiniBand connections between prompt and token machines, and heterogeneous Splitwise-HA may not be readily available with such connectivity.Source: §VII · p. 12

Related systems

  • Existing model-serving approaches generally use the same machine for prompt and token phases, whereas Splitwise separates them.Source: §VIII · p. 12

APPENDIX

The released artifact provides traces, a KV-cache transfer prototype, and simulation components for evaluating Splitwise and future LLM serving systems.

Artifact contents

  • Production traces from two Microsoft Azure LLM inference services are included in the artifact.Source: A. Abstract · p. 14
  • A prototype implementation of Splitwise’s KV-cache transfer mechanism in vLLM is included.Source: A. Abstract · p. 14
  • SplitwiseSim is a discrete-event simulator for evaluating model serving in LLM inference clusters.Artifact functionality was tested only for the traces and SplitwiseSim because of limited hardware availability.Source: A. Abstract · p. 14

Runtime and hardware

Availability

C. Description

The artifact is distributed through Zenodo and several online repositories, with components covering traces, KV-cache transfer, and SplitwiseSim. It documents hardware and software dependencies for the transfer prototype and simulation environment.

Artifact Availability

  • The complete artifact is available as a Zenodo archive, with individual components distributed through separate online repositories.The listed components include production traces, a KV-cache transfer prototype, and SplitwiseSim with experiment and plotting scripts.Source: C. Description · pp. 14–15

Hardware Dependencies

  • The KV-cache transfer prototype requires two GPU machines connected over Infiniband.Examples include NVIDIA DGX-A100 or DGX-H100 systems.Source: C. Description · p. 15
  • SplitwiseSim runs on a standard x86-64 CPU machine, with multiple machines available to parallelize simulation runs.Source: C. Description · p. 15

Software Dependencies

  • The transfer prototype builds on vLLM and MSCCL++, while SplitwiseSim uses publicly available Python packages installed through requirements.txt.Source: C. Description · p. 15

Data Sets

  • Coding and conversation traces from Microsoft Azure are included online as part of the artifact release.Source: C. Description · p. 15

Workflow

Related papers

Loading 2311.18677v2…