LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization

1 Introduction

• Heterogeneous production clusters contain fewer high-caliber GPUs and many lower-caliber inference GPUs, whose utilization can differ substantially.Better use of these available GPUs could reduce provisioning costs for LLM-enabled applications.Source: §1 · pp. 1–2
• Uniform tensor- and pipeline-parallel partitioning can underutilize high-capacity GPUs or cause out-of-memory errors on low-memory GPUs.Decoder-only LLM serving also has prefill and decode phases with workload-dependent execution times, motivating phase-aware partitioning.Source: §1 · p. 2
• Uniform quantization can waste memory on large GPUs or force low-precision weights on all layers, whereas adaptive mixed precision can match GPU capacities and improve model quality.The paper identifies adaptive quantization as preferable for heterogeneous serving.Source: §1 · p. 2
• LLM-PQ jointly selects layer partitioning, quantization precision, and hybrid micro-batch sizing for a given heterogeneous cluster and workload.Its search combines GPU-ordering and phase–micro-batch exploration with integer linear programming, supported by cost modeling and layer-sensitivity estimates.Source: §1 · p. 2
• 2.88× maximum and 2.26× average throughput improvements were achieved across 11 heterogeneous GPU clusters over state-of-the-art approaches.The prototype also produced cost-model predictions with less than 6% error.Source: §1 · p. 2

2.1 Generative Inference of LLM

• Prefill processes the prompt once to build KV caches, whereas decode generates tokens sequentially using updated KV pairs.Source: §2.1 · pp. 2–3
• Prompt length substantially changes phase costs: prefill is at least 36% of execution for long prompts, while decode depends on generated-token count.Source: §2.1 · p. 3

2.2 Heterogenous Model Parallelization

• 14.53× versus 7.29×: under FP16, P100 is respectively slower than V100 for one layer’s prefill and decode phases, requiring phase-aware partitioning.Source: §2.2 · p. 3
• Uniform partitioning and precision can underutilize high-memory GPUs or cause OOM on low-memory GPUs, motivating heterogeneous layer placement and adaptive mixed precision.Source: §2.2 · p. 3, §2.4 · p. 4

2.3 Online and Offline Serving Task

• LLM-PQ targets offline serving, where prompt length and token-generation count are known in advance, unlike online workloads with unpredictable requests.Source: §2.3 · p. 3

2.4 Quantization in LLM

• Mixed-precision quantization can improve performance over uniformly using the lower bit, while low precision may not accelerate inference because quantization overhead varies by device and workload.Source: §2.4 · p. 4, Figure 5 · p. 4

2.5 Challenges

• LLM-PQ jointly considers precision, layer partition, micro-batch size, prompt length, token count, and GPU type, using cost models and an efficient pruning algorithm to address the large search space.Source: §2.5 · pp. 4–5

• The offline assigner jointly selects model layer partitions, micro-batch sizes, and per-layer quantization bitwidths from workload, device, precision, and quality inputs.Source: §3 · p. 5
• Analytical memory and latency models estimate shard resource use and execution time, while an indicator generator measures performance perturbation from layer quantization.Source: §3 · p. 5
• The distributed runtime executes the assigned plan, with a master engine handling token-processing coordination and workers running pipeline stages on separate GPUs.Source: §3 · p. 5

4.1 Cost Model

• The cost model accounts for model weights, KV cache, and temporary memory, while modeling prefill latency from FLOPs and decode latency from memory operations.It profiles decoder-layer execution under different precisions, prompt lengths, and batch sizes, then fits linear regression models for each phase.Source: §4.1 · pp. 5–6

4.2 Indicator of Model Perturbation by Quantization

• The variance indicator ranks layer sensitivity to quantization by estimating quantization-induced output variance across linear operators.It uses elementwise mean and variance operations, reducing computation complexity relative to Hessian-based sensitivity estimation.Source: §4.2 · pp. 6–7

4.3 Optimizer

• Algorithm 1 enumerates device orderings and micro-batch sizes, then solves an ILP to assign each layer’s device and quantization bitwidth while respecting memory and continuous-partition constraints.The formulation enforces one bitwidth and one device per layer, device memory limits, embedding placement, and adjacent-stage layer partitioning.Source: §4.3 · pp. 7–8
• The optimizer minimizes pipeline serving latency and model-quality degradation, with θ controlling the trade-off between acceleration and quality.End-to-end latency combines prefill and decode execution, including micro-batch pipeline effects and the generated-token sequence.Source: §4.3 · p. 7
• Search-space optimizations prune micro-batch candidates, group layers for larger models, and use bitwidth transfer to iteratively improve slow pipeline stages.Bitwidth transfer can exchange precision and layer counts between stages, particularly when KV size does not dominate memory occupation.Source: §4.3 · p. 8

• The prototype extends HuggingFace models with preallocated KV caches, adaptive quantization, heterogeneous pipeline serving, and thread-safe micro-batch management.Asynchronous communication is used among pipeline stages.Source: §5 · pp. 8–9
• On-the-fly quantized loading overlaps disk-to-CPU loading, GPU quantization, and CPU-to-GPU copying to reduce DRAM requirements and improve recovery speed.Source: §5 · p. 9
• LLM-PQ provides commands to generate and launch deployment strategies for specified models, devices, workloads, and optimization settings.Strategies can be registered automatically when identical GPU types share a node, or launched with distributed configurations.Source: §5 · p. 9

6.2 Fidelity of Cost Models

• The cost models closely match real systems: memory prediction error is almost negligible, while average latency prediction error is less than 6%.Prefill cost generally grows linearly with workload, whereas decode latency changes notably only after context-length increases of 50–100.Source: §6.2 · p. 10, Figure 7 · p. 10

6.3 Serving in Heterogeneous Clusters

• LLM-PQ achieves the highest inference throughput and best model accuracy in clusters 3, 4, 6, 7, and 8, while outperforming major serving baselines in most cases.In cluster 2, it achieves 1.5× faster inference with only a 0.01 perplexity drop; in cluster 6, its perplexity surpasses FP16.Source: §6.3 · p. 10, Table 4 · p. 11

6.4 Serving in Homogeneous Clusters

• On homogeneous clusters, LLM-PQ still gains throughput, but gains are smaller than on heterogeneous clusters.Cluster 9 is an exception where FlexGen-int8 has better performance and perplexity because limited memory causes heavier compression and more low-precision kernels.Source: §6.4 · p. 10, Table 5 · p. 11

6.5 Effectiveness of Variance Indicator

• The variance indicator improves perplexity over random assignment, while Hessian-based and LLM-PQ indicators match under heavier quantization on cluster 9.Source: §6.5 · p. 10, Table 6 · p. 11

6.6 Serving with Shorter Prompts

• With shorter prompts and 200 generated tokens, LLM-PQ provides substantial speedups without accuracy degradation and sometimes improves accuracy.Its lower gain in cluster 4 is attributed to reduced KV-cache memory and a workload closer to a one-phase system.Source: §6.6 · p. 10, Table 7 · p. 11

6.7 Approaches Expediting Optimizer Algorithm

• Grouping reduces optimizer complexity, while heuristic search can achieve the best throughput with the smallest overhead in clusters 4 and 10.Full grouping can produce better solutions but incurs greater overhead and may fail to find a good solution within the 60-second limit.Source: §6.7 · pp. 10–11, Table 8 · p. 11

6.8 Parameter Sensitivity

• Increasing the quality scalar generally lowers inference throughput and raises model accuracy because optimization places less weight on latency.Source: §6.8 · p. 12, Figure 8 · p. 12

6.9 Comparison with Pure Adaptive Quantization

• LLM-PQ outperforms pure adaptive quantization with adabits in all selected workload and cluster cases.Source: §6.9 · p. 12, Figure 9 · p. 12

• Tensor parallelism is not implemented, but the paper argues it can be incorporated by enumerating device meshes and treating tensor-parallel groups as composite devices.Source: §7 · p. 12
• Newer quantization schemes such as AWQ, SpQR, and QLoRA are presented as candidate methods that can be integrated into LLM-PQ.Source: §7 · p. 12
• Applying LLM-PQ to online systems such as ORCA or vLLM requires new design considerations because workload and available KV memory are unpredictable.The discussion highlights a trade-off between quantized-operator speed and memory available for KV storage.Source: §7 · p. 12

• LLM-PQ jointly optimizes precision assignments, layer partitions, micro-batch sizes, and latency across token-generation phases.Source: §8 · p. 12
• Its cost models predict memory occupation and execution latency for mixed-precision LLM serving, while an indicator guides bitwidth selection.Source: §8 · p. 12
• Experiments on varied cluster setups show that LLM-PQ surpasses state-of-the-art approaches for heterogeneous LLM serving.Source: §8 · p. 12

• Weight-only quantization introduces output variance proportional to weight dimension, scaling factors, and input variance.The scalar multiplication is modeled as y = ẇx, with quantized weights ẇ ∈ Q(W) and inputs sampled from distribution D.Source: §A.1 · p. 15
• Under deterministic rounding, a quantized scalar takes the stated rounded form, and its error err_w = w − ˜w is deterministic and bounded by ±1.Source: §A.1 · p. 15
• Under deterministic rounding, output variance is expressed through the input variance and the quantization-error contribution.The supplied derivation includes a term involving the weight dimension, scaling factor, and Var[X].Source: §A.1 · p. 15
• Stochastic rounding preserves the expected quantized weight while decomposing output variance into weight-, input-, and quantization-variance terms.The passage states E[Ẇ] = E[W] and presents a variance expression involving E[Ẇ]^2 Var[X], E[X]^2 Var[Ẇ], and an additional term.Source: §A.1 · p. 15

A.2.1 𝜃and Solver Setup.

• Table 9 provides the θ and solver configurations used for both heterogeneous and homogeneous LLM-PQ results.Source: §A.2.1 · p. 15
• 31s solving time is reported for OPT66B on a three-node cluster containing two P100, two V100, and two A100 GPUs.The solving time uses the heuristic configuration.Source: §A.2.2 · p. 15