PaperlayerView exact revision on arXiv

Source-linked AI summary

Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads

Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, Yizhou Shan

arXiv:2401.11181v1cs.DC

TL;DR

Heterogeneous LLM inference requests interfere because prefill and decode have distinct computational and memory characteristics. TetriInfer addresses this by chunking prompts, disaggregating prefill and decode, and scheduling requests using predicted resource usage. It outperforms vanilla vLLM on TTFT, JCT, and performance per dollar across most common workloads, while its benefits are marginal for heavy-prefill and heavy-decode workloads.

  • Problem

    Heterogeneous prompt and output lengths create interference when downstream workloads share LLM inference infrastructure.

  • Method

    TetriInfer groups and schedules requests by characteristics using fixed-size prefill chunks, disaggregated prefill and decode instances, and a two-level resource-aware decode scheduler.

  • Results

    TetriInfer outperforms vanilla vLLM in TTFT, JCT, and performance per dollar across most common workloads.

  • Takeaways & Limitations

    Characteristic-aware scheduling and phase disaggregation provide effective LLM inference serving for common mixed workloads.

  • Takeaways & Limitations

    TetriInfer is not ideal for heavy-prefill and heavy-decode workloads, and the implementation uses request-level rather than chunk-level KV-cache transfer.

Abstract

from arXiv · show

Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in TetriInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computationsaturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that TetriInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in turns of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.

1 Introduction

LLM inference workloads vary substantially in prompt and output lengths, and co-running these heterogeneous requests creates serious interference. TetriInfer addresses this by grouping and scheduling requests according to their characteristics, using chunked prefill, disaggregated phases, and resource-aware decode scheduling.

  • LLM inference generates the first token during prefill, then produces output tokens autoregressively during decode.
  • Prompt and generated-token lengths vary dramatically across downstream tasks, with summarization using long prompts and short outputs while content creation shows the opposite pattern.Token lengths across tasks can differ by more than two orders of magnitude.
  • Mixing heterogeneous requests causes severe interference: prefill mixing can slow requests by more than 10x, prefill-decode mixing by 5x, and unequal decode mixing cuts throughput by 16%.Static per-task provisioning is impractical because LLM serving infrastructure is expensive.
  • TetriInfer partitions prompts into fixed-size chunks so accelerators operate near their computation-saturated limit during prefill.The design limits tokens processed in one prefill iteration and pads prompts into fixed-size chunks.
  • TetriInfer disaggregates prefill and decode into independently scalable instances, transferring prefilled KV caches between them.The instances can also flip roles when workload changes.
  • A two-level scheduler uses predicted generated-token lengths and resource usage to reduce decode scheduling hotspots.TetriInfer incorporates an LLM-based length predictor to schedule decode requests accordingly.
  • Against vanilla vLLM, TetriInfer improves TTFT, JCT, and performance per dollar across most common workloads.For common mixed workloads, average TTFT improves by 85% and average JCT by 50%; light-prefill, heavy-decode workloads see 2.4x higher performance per dollar.

2 Background and Motivation

LLM inference separates into computation-bound prefill and memory-bound decode phases, but mixed workloads interfere severely because their prompt and generation lengths vary widely. Experiments show substantial slowdowns across prefill–prefill, prefill–decode, and decode–decode combinations, motivating characteristic-aware scheduling and grouping.

  • 2.1 Generative LLM Inference: LLM inference has computation-bound prefill and memory-bound decode phases, with throughput limited by accelerator computation or memory bandwidth.Prefill generates the first token and KV cache; decode generates subsequent tokens autoregressively.
  • 2.2.1 Prefill and Prefill: More than 10x prefill latency slowdown occurs when light or heavy prefill requests co-run, while exceeding the accelerator-saturate threshold sharply increases prefill latency.Light-prefill latency also rises 2x with 7 concurrent light requests and 8x with 63.
  • 2.2.2 Prefill and Decode: 5x decoding-latency growth occurs when one heavy prefill request joins a light decode batch, while mixed prefill latency rises by up to 2.5x.The prefill latency increases once more than 7 light decode requests co-run.
  • 2.2.3 Decode and Decode: Decode batches containing more heavy requests suffer lower throughput and higher latency than batches containing only light decode requests.The cited experiment compares different mixtures at a batch size of 128.
  • 2.3 Analysis and Insights: The proposed remedy is to cap prefill work per step, disaggregate prefill and decode, and use predicted resource usage in two-level scheduling to avoid decode hotspots.These designs target computation saturation, phase interference, and decode contention, respectively.

3.1 Overview

TetriInfer organizes inference around dedicated prefill and decode instances, fixed-size prefill computation units, and predicted decode resource usage. Its modules coordinate scheduling, prediction, execution, and dispatch across the separated phases.

  • 3.1 Overview: TetriInfer runs prefill and decode on dedicated instances rather than mixing both phases on the same instance.Requests are scheduled to the instance type corresponding to their current phase.
  • 3.1 Overview: Fixed-size, padded prompt chunks keep prefill execution close to the accelerator’s computation-saturated limit.The design limits tokens processed in one prefill computation unit.
  • 3.1 Overview: The architecture highlights a centralized control plane, prefill instances, decode instances, and a length prediction model as four core modules.These modules organize cluster management, phase-specific execution, and resource-aware scheduling.
  • 3.1 Overview: Each prefill instance includes a local scheduler, length predictor, LLM engine, and dispatcher that process requests through sorting, prediction, prefill, and dispatch.The supplied passage describes the first three processing components and begins the fourth.
  • 3.1 Overview: Decode instances accept requests from any prefill instance and use local scheduling policies to select decode work.This enables decode execution to remain separated from prefill execution.
  • 3.1 Overview: A fine-tuned small LLM predicts generation lengths offline so scheduling can estimate decode resource usage and avoid hotspots.The predictor is deployed at every prefill instance and informs both dispatch and decode scheduling.

3.2 Control Plane

TetriInfer’s control plane manages inference requests and cluster resources through a global scheduler and cluster monitor. It forwards requests to lightly loaded prefill instances, broadcasts load information, and supports lifecycle management without a single processing bottleneck.

  • 3.2 Control Plane: The centralized control plane comprises a cluster monitor for instance lifecycle and a global scheduler for inference-request lifecycle.It is described as a distributed system without a single point of failure or processing bottleneck.
  • 3.2 Control Plane: The cluster monitor collects load statistics from prefill and decode instances and broadcasts aggregated decode load to prefill instances.Instances regularly report load information, such as every 100 ms.
  • 3.2 Control Plane: The global scheduler forwards incoming requests to prefill instances and streams inference outputs back to external services.It also maintains request status information including arrival time, current phase, and SLA requirement.
  • 3.2 Control Plane: TetriInfer’s timeline separates prefill and decode instances, allowing long decoding tasks to be load-balanced across on-demand decode instances.The workflow compares this arrangement with systems that run mixed prefill and decode on two nodes.
  • 3.2 Control Plane: When a request arrives, the global scheduler selects the least-loaded prefill instance and records the request in its status table.The scheduler chooses only the prefill instance; the prefill dispatcher selects decode instances using predicted resource usage.

3.3 Prefill Instance

The prefill instance combines request scheduling, length prediction, chunked prefill, KV-cache dispatch, and network abstraction to reduce interference and coordinate downstream decoding.

  • 3.3.1 Prefill Scheduler: FCFS, SJF, and LJF schedule prefill requests, with SJF and LJF ordering requests by prompt-token length.FCFS preserves arrival order; SJF and LJF sort requests in ascending and descending prompt length, respectively.
  • 3.3.1 Prefill Scheduler: A scheduling batch limits how many requests are sorted at once, preventing starvation under SJF and LJF.For example, a batch size of ten schedules twenty queued requests in two sorted groups.
  • 3.3.1 Prefill Scheduler: 7.8% lower average prefill waiting time is achieved by SJF versus FCFS with batch size 16.The improvement becomes larger with larger batch sizes.
  • 3.3.2 Length Predictor: 74.9% accuracy is achieved when predicting generated-token length ranges with 200-token granularity.Length ranges guide scheduling through estimated resource-usage bounds rather than exact token counts.
  • 3.3.3 Chunked Prefill: 86.4% lower average prefill latency is obtained by chunked prefill with FCFS versus vanilla vLLM.Prompts are sliced and merged into fixed-size chunks, with partial final chunks padded to ChunkSize.
  • 3.3.4 Dispatcher: The dispatcher selects decode instances for prefilled requests and transfers their KV caches through a unified network abstraction.The design supports different network-stack options; this work implements request-level KV-cache transfer and leaves chunk-level transfer for future work.

3.4 Decode Instance

The decode instance receives prefilled KV caches, queues requests with continuous batching, and uses predicted working-set sizes to avoid memory thrashing.

  • 3.4 Decode Instance: The decode instance receives remote KV caches before adding requests to its local scheduler queue.Its processing path includes a network receiver, local scheduler, and decoding LLM.
  • 3.4 Decode Instance: vLLM’s greedy scheduling can exhaust memory in future iterations and cause thrashing because it ignores working-set size.The limitation arises when requests are admitted while spare memory exists in the current iteration.
  • 3.4 Decode Instance: Reserve-static admits requests only when predicted current-iteration memory usage fits available accelerator memory.Reserve-dynamic additionally checks whether memory remains available after the shortest remaining job finishes.
  • 3.4 Decode Instance: Reserve-dynamic uses predicted remaining-token demand to mitigate memory thrashing while retaining paging benefits.Both reserve policies are designed to prevent swaps.

3.5 Instance Flip

TetriInfer dynamically flips prefill and decode instances so their ratio can respond to workload variation within fixed hardware resources.

  • 3.5 Instance Flip: Dynamic instance flipping adjusts the prefill-to-decode ratio because inference workloads vary substantially in their prefill and decode needs.TetriInfer can also scale out by allocating additional hardware resources.
  • 3.5 Instance Flip: A transition watcher regularly checks instance load and applies configurable flipping policies.One example flips an instance after its load remains below 10% for one minute.
  • 3.5 Instance Flip: Prefill flipping stops new requests, drains queued work, and then changes the instance role.Flipping a decode instance requires notifying prefill instances to stop forwarding requests to it.

4 Implementation

The implementation uses Python and C++ components around vLLM-based prefill and decode instances, while emulating network bandwidth because high-end hardware was unavailable.

  • 4 Implementation: The centralized control plane is implemented in Python, while the unified network stack uses C++ and IB Verbs for transfer.Shared-memory communication enables command transfer between the Python and C++ components.
  • 4 Implementation: Each deployable prefill or decode instance contains separate Python and C++ processes.Prefill Python handles scheduling, prediction, and the main LLM; decode Python handles scheduling and decoding, while C++ runs networking.
  • 4 Implementation: The implementation supports only indirect socket-based transfer and emulates varying bandwidth instead of transferring actual KV caches during tests.Prefilled KV caches are loaded into decode memory beforehand, and transfer latency is calculated from model architecture and target bandwidth.

5 Evaluation

TetriInfer is evaluated against vanilla vLLM across diverse workloads using TTFT, JCT, resource usage time, and performance per dollar. Results show substantial gains for light-prefill workloads, while heavy-prefill workloads involve resource trade-offs and scheduler studies validate chunking, prediction, and load balancing.

  • End-to-End Performance: The evaluation compares TetriInfer with vanilla vLLM across HPLD, HPHD, LPHD, LPLD, and mixed workloads using TTFT, JCT, and resource usage time.Perf/$ is also reported, and resource usage aggregates prefill and decode instance wall time.
  • Light Prefill and Light Decode: 44% lower average TTFT and 40% lower average JCT are achieved on LPLD, with comparable resource usage and 1.4x higher perf/$.This workload uses 128 requests and represents chat workloads.
  • Light Prefill and Heavy Decode: 97% lower average TTFT, 47% lower average JCT, and 38% less hardware resource usage are achieved on LPHD, improving perf/$ by 2.4x.The reported gains are attributed to disaggregated prefill and decode and variable decode batch sizing.
  • Heavy Prefill Workloads: HPLD improves average TTFT and JCT by 9% and 23% but increases resource usage by 43%, while vLLM leads perf/$ by 14%.For HPHD, TetriInfer improves JCT by 19% with 7% more resources and raises perf/$ by 1.1x.
  • Mixed: Mixed workloads reduce average TTFT, JCT, and resource usage by 85%, 50%, and 21%, respectively, while improving perf/$ by 1.9x.The mixed test runs 128 randomly sampled ShareGPT requests.
  • Scheduler Policy and Batch Size: Chunked prefill with FCFS lowers latency by 86.4% versus vLLM’s fixed batch mode, while SJF further reduces average waiting time by 7.8%.The comparison uses ShareGPT-distributed requests with PrefillSchedBatch set to 16.
  • Length Predictor: Co-running OPT-13B with the OPT-125M predictor increases average prefill latency by 10% and reduces throughput by 12% in stress tests.With a 512-token padding limit, 80% of large-model prefill requests remain unchanged; the authors state practical impact will be smaller.
  • Intra-Decode Scheduling: Reserve-dynamic scheduling matches vLLM’s greedy JCT at 74.9% prediction accuracy, while ideal predictions achieve the lowest total decoding time.The inter-decode scheduler balances heavy- and light-decode load across instances.

6 Related Work

TetriInfer complements prior efficiency techniques by addressing interference through prefill-decode disaggregation and chunked prefill. Related systems include concurrent disaggregation work and chunked-prefill approaches with different execution organization.

  • Disaggregation: TetriInfer is among the first systems to disaggregate prefill and decode, concurrently with Splitwise.The comparison positions TetriInfer within closely related LLM inference-serving systems.
  • Chunked Prefill: Sarathi uses chunked prefill with mixed prefill-decode chunks, whereas TetriInfer uses a different execution organization.The passage contrasts their approaches but is truncated after introducing TetriInfer’s design.
  • Orthogonal Techniques: Quantization, paging, and low-level kernel optimizations are orthogonal to TetriInfer’s effort to mitigate interference.These techniques target model-weight precision, memory fragmentation, or kernel efficiency rather than the interference problem described here.

7 Conclusion

TetriInfer addresses LLM inference interference by scheduling and grouping requests according to their characteristics. Its fixed-size prompt chunks, disaggregated prefill and decode instances, and two-level scheduling improve TTFT, JCT, and performance per dollar.

  • Conclusion: TetriInfer partitions prompts into fixed-size chunks to keep the accelerator near its computation-saturated limit.This is the system’s first design component for reducing prefill interference.
  • Conclusion: TetriInfer disaggregates prefill and decode instances to avoid interference when the phases run together.The dedicated instances allow the two phases to operate separately.
  • Conclusion: A two-level scheduling algorithm avoids decode scheduling hotspots by organizing requests according to their characteristics.The conclusion identifies this as the third key system component.
  • Conclusion: TetriInfer improves TTFT, JCT, and performance per dollar by a large margin.These are the paper’s reported headline efficiency and latency outcomes.
Loading 2401.11181v1…