Source-linked AI summary
Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
Javad Mirzaei, Jeebak Mitra
TL;DR
LLM serving must balance phase-specific latency, throughput, and workload-dependent parallelism choices. The paper maps TTFT–ITL and SLO trade-offs and finds that preferred configurations shift with workload balance, latency strictness, and context length.
Problem
Serving must balance prefill-related TTFT, decode-related ITL, throughput, and GPU use, while parallelism effectiveness varies with workload and latency objectives.
Source: §1 · pp. 1–2Method
The paper constructs TTFT–ITL feasible sets and Pareto frontiers, formulates goodput-based SLO selection, and analyzes long-context scalability.
Source: §1 · p. 2Results
TP-dominant configurations favor decode-heavy workloads and relaxed TTFT SLOs, while PP-dominant configurations favor prefill-heavy workloads, stringent TTFT SLOs, and longer contexts.
Source: §1 · p. 2, §5 · p. 13Takeaways & Limitations
Parallelism should be treated as a workload-, SLO-, and context-dependent operating decision rather than a fixed deployment choice.
Source: §5 · p. 14The Pareto-frontier analysis fixes total sequence length to isolate workload composition; changing sequence scale requires a new feasible set and frontier.
Source: §2.1 · p. 5
Abstract
from arXiv · showhide
Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources. This challenge arises because inference consists of two fundamentally different phases: prefill, which determines time-to-first-token (TTFT), and decode, which determines inter-token latency (ITL). The use of tensor parallelism (TP), pipeline parallelism (PP), and hybrid configurations exposes different trade-offs between these latency dimensions, making the optimal parallelism configuration highly dependent on workload characteristics, latency requirements, and context length. Moreover, despite the widespread adoption of distributed inference, there is still limited understanding of how parallelism configurations should be selected under varying workload conditions and service-level objectives (SLO). In this paper, we develop a systematic framework for characterizing LLM inference parallelism via throughput-latency trade-offs, SLO-aware serving, and long-context scalability. We introduce the notion of parallelism operating regions by constructing TTFT-ITL feasible sets and Pareto frontiers across different workload compositions, thereby revealing a systematic transition from TP-dominant to PP-dominant configurations as workloads become increasingly prefill-heavy. We further formulate SLO-aware parallelism selection using goodput, showing that stringent TTFT requirements favor PP-dominated configurations, whereas relaxed latency constraints progressively shift the optimality toward TP-dominated configurations with higher serving capacity. Finally, we demonstrate that increasing context length fundamentally changes the relative advantages of parallelism configurations, with PP-dominant configurations exhibiting more favorable scalability and SLO attainment in the long-context regime.
1 Introduction
LLM serving must balance phase-specific latency, throughput, and GPU use, while workload and latency conditions make parallelism a workload-dependent choice. The paper situates this challenge among work on latency-aware goodput, scheduling, and long-context serving.
1 Introduction
- Prefill and decode have distinct characteristics, creating a trade-off between TTFT and ITL alongside throughput and GPU-utilization demands.The introduction argues that one latency or throughput metric alone is often insufficient for evaluating serving performance.Source: §1 · p. 1
- No TP, PP, or hybrid configuration is universally optimal; effectiveness depends on model size, sequence length, hardware topology, workload composition, and latency objectives [7] [8] [9].The paper frames parallelism as a workload-dependent operating point rather than a static deployment decision.Source: §1 · p. 2
- The paper studies parallelism through TTFT–ITL feasible sets and Pareto frontiers, SLO-aware selection using goodput, and long-context scalability.It reports a shift from TP-dominant to PP-dominant configurations as workloads move from decode-heavy to prefill-heavy regimes.Source: §1 · p. 2
1.1 Related Works
- Prior work motivates evaluating latency compliance alongside throughput, since request characteristics and SLOs shape useful serving performance [18] [19].Goodput counts throughput from requests that meet latency requirements.Source: §1.1 · pp. 2–3
- Related work connects parallelism choices with scheduling and resource allocation, including chunked prefill for interleaving decode and serving heterogeneous SLOs [20] [22].These studies address prompt interference, differentiated latency priorities, and uncertain output-length demands.Source: §1.1 · pp. 2–3
- Long-context workloads increase prefill attention computation and KV-cache requirements; context parallelism distributes context tokens and state across accelerators, but communication costs can outweigh benefits for shorter-context workloads [16] [28] [29] [30] [31] [32] [33].Source: §1.1 · p. 3
2 Parallelism Operating Region
The section models parallelism as workload-dependent TTFT–ITL operating regions, then shows how workload composition and output length shift the available trade-offs.
2 Parallelism Operating Region
- The framework represents parallelism as p = (NTP, NPP) and workload as w = (Lin, Lout, C), while retaining TTFT and ITL as separate latency dimensions.The two measures capture complementary latency behavior and can differ substantially even when total request latency is similar.Source: §2 · p. 4
- TP can reduce per-GPU computation but raises communication costs; PP avoids much fine-grained collective communication but can incur scheduling and pipeline-bubble overhead.These costs affect prefill and decode differently, shaping their complementary latency behavior.Source: §2 · p. 4
2.1 Latency Feasibility Set and Pareto Frontier
- Fixing concurrency and total sequence length lets input/output composition vary while keeping workload scale approximately constant, isolating prefill–decode balance.The feasible set and its Pareto frontier are workload-dependent, rather than a single global ranking of configurations.Source: §2.1 · p. 5
2.2 Experiment Results on Pareto Frontier
- As workloads move from decode-heavy to prefill-heavy, the Pareto-optimal configuration shifts from TP-dominant through hybrid to PP-dominant.TP8 favors low ITL, PP8 favors low TTFT, and the hybrid configurations bridge the extremes.Source: Figure 1 · p. 6, §2.2 · p. 7
- TP8 occupies the low-ITL/high-TTFT end, while PP8 occupies the low-TTFT/high-ITL end; hybrids provide intermediate trade-offs.TP distributes layer computation but incurs collective communication; PP avoids much of that communication, while pipeline scheduling and bubbles can constrain decode.Source: §2.2 · pp. 6–7
- Longer output sequence length mainly expands the frontier along ITL, with the largest ITL benefit for TP-dominant configurations and little TTFT change.The reported comparison varies Lout across 200, 400, and 800 while holding input length and concurrency fixed.Source: Figure 2 · p. 7, §2.2 · p. 8
3 SLO-Aware Parallelism
The section selects parallelism by maximizing goodput, which balances request throughput with TTFT-SLO compliance. In the reported workload, the goodput-optimal configuration shifts from PP8 to hybrid and then TP8 as the TTFT SLO is relaxed.
3 SLO-Aware Parallelism
- Goodput is request throughput multiplied by the fraction of requests meeting the TTFT SLO, so selection balances serving capacity against latency compliance.The analysis considers TTFT as its SLO constraint.Source: §3 · pp. 8–9
3.1 Experiment Results on SLO-Aware Parallelism Selection
- PP-dominant configurations favor prefill and lower TTFT, whereas TP-dominant configurations favor decoding and lower ITL; hybrid behavior falls between these extremes.PP8's favorable TTFT distribution is linked to avoiding TP's fine-grained collective communication, while TP8's decoding advantage reflects lower per-token computation time across GPUs.Source: §3.1 · pp. 9–10, Figure 3 · p. 9
- PP8 has the highest goodput at stringent TTFT SLOs, while progressively relaxed SLOs shift the optimum through hybrid configurations to TP8.The reported comparison uses SLOs of 5 and 10 seconds as stringent, then 20 and 25 seconds as progressively relaxed.Source: Figure 4 · p. 10, §3.1 · pp. 10–11, Figure 5 · p. 11
- The trade-off is between throughput and TTFT attainment: PP-dominant configurations tend to improve compliance at the expense of throughput, while TP-dominant configurations generally do the reverse.As the SLO becomes sufficiently loose, compliance approaches one and goodput approaches raw throughput, favoring configurations with greater serving capacity.Source: §3 · p. 9, §3.1 · p. 10
4 Parallelism for Long Context
The section examines how increasing input length changes the feasible performance of parallelism configurations while holding output length fixed. As prefill grows, PP-dominant configurations scale more favorably than TP-dominant ones, particularly under tight TTFT requirements.
4 Parallelism for Long Context
- The experiment increases input sequence length while keeping output length fixed to isolate how context growth changes throughput, latency, SLO compliance, and feasibility.Longer inputs increase prefill computation, TTFT, and memory required for intermediate states and the KV cache.Source: §4 · p. 11
- As sequence length increases, PP-dominant configurations become more competitive and can achieve higher goodput than TP-dominant configurations, especially under tighter TTFT SLOs.The comparison holds output length fixed; TP's shorter-context serving-capacity advantage is progressively eroded as prefill grows.Source: Figure 6 · p. 12, §4 · p. 12
- PP-dominant configurations retain higher SLO attainment as sequence length grows, while TP8 declines faster, particularly under the tighter SLO.The decline in SLO attainment is sharper at Ts = 2(s) than at Ts = 5(s).Source: §4 · pp. 12–13, Figure 7 · p. 13
- Goodput depends on both serving throughput and SLO attainment, so a faster decline in compliance can overwhelm TP-dominant configurations’ throughput advantage.This explains why PP-dominant configurations can sustain useful throughput farther into the long-context regime.Source: §4 · pp. 12–13
5 Conclusions
The paper concludes that parallelism choices occupy workload-dependent points in the TTFT–ITL design space, rather than one universally optimal configuration. Its findings connect workload composition, SLO requirements, and context length to which configurations are advantageous.
5 Conclusions
- Parallelism configurations form a workload-dependent Pareto frontier, shifting from TP-dominant for decode-heavy workloads to PP-dominant for prefill-heavy workloads.TTFT and ITL should be treated as complementary dimensions because average latency alone is insufficient to characterize serving performance.Source: §5 · p. 13
- Stringent TTFT requirements favor PP-dominant configurations, while relaxed SLOs shift optimality toward TP-dominant configurations with higher throughput capacity.The selection balances serving capacity with latency compliance through goodput.Source: §5 · p. 13
- As context length increases, PP-dominant configurations exhibit more favorable scaling and maintain higher SLO attainment in the long-context regime.The paper frames parallelism as a workload-, SLO-, and context-dependent operating decision.Source: §5 · pp. 13–14