PaperlayerView exact revision on arXiv

Source-linked AI summary

More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving

Srikanta Datta Tumkur, Mehar Simhadri, Anshu Bansal, Jay Iyer, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly, Raj Dandekar

arXiv:2608.23962v1cs.AI

TL;DR

The paper asks whether adding GPUs or compressing the KV cache is cheaper for memory-bound LLM serving, a comparison missing from the separate scaling and compression literatures. It evaluates both on a cost-normalised frontier using a profiled simulator, finding no crossover: compression is cheaper when it can fit, while tensor parallelism is required when weights do not fit.

  • Problem

    Tensor parallelism and KV compression both provide memory headroom, but their costs are rarely compared on a common axis for fixed model, quality, and latency requirements.

  • Method

    The study sweeps tensor-parallel degrees and KV-compression settings across models, workloads, and GPUs, evaluating them with a profiled simulator on cost per million tokens and latency.

  • Results

    Compression is cheaper than tensor parallelism at every matched memory-relief level, with margins from 1.20× to 2.00× across tested hardware.

  • Takeaways & Limitations

    Compression is a capacity mechanism, whereas tensor parallelism is the entry requirement when model weights exceed one device’s memory.

  • Takeaways & Limitations

    Quality was not evaluated, so compression results are upper bounds valid only while the quality floor permits each setting.

Abstract

from arXiv · show

When an LLM serving deployment runs out of KVcache room, there are two well-established ways out. Tensor parallelism shards the weights and the KV cache across two, four, or eight devices, buying memory headroom at the price of an all-reduce on every layer and a hardware bill that grows with the device count. The algorithms community shrinks the cache in place, with KV quantisation and eviction keeping a single GPU and spending a little quality instead. Compression papers report memory ratios, parallel-scaling papers report throughput curves, and almost nobody puts the two on the same cost axis. We place tensor-parallel configurations (degree 1 to 8) and KV-compressed configurations (16/8/4-bit, keep-ratios down to 0.25) on one costnormalised axis, cost per million tokens against latency, using a profiled simulator calibrated on A100, A40, and H100 hardware, and we go looking for the cost-equivalence crossover. We do not find one. Across two models (Llama-2 at 7B and 70B), three GPU types, and every level of memory relief we could construct, compression is cheaper by 1.20x to 2.00x. A 7B model on an 80 GB device cannot exhaust its KV budget within its own context window, and the boundary that decides between the strategies is model size relative to device memory, at roughly 36B parameters for an 80 GB card. Below that wall, compression dominates and extra GPUs are largely wasted spend; above it, tensor parallelism stops being a choice and becomes an entry ticket: Llama-2-70B is infeasible on one A100 at any KV setting, because the binding resource is weights, which KV compression does not touch. Tensor parallelism is the only lever that improves latency (compression makes per-token latency worse, by 8 to 93%, through batching contention), while compression is the only lever that multiplies capacity per dollar (16.5x, against 1.21x for an eightfold spend on GPUs).

I. INTRODUCTION

Memory-bound LLM serving can add GPUs through tensor parallelism or shrink the KV cache through quantisation and eviction. The paper places both strategies on a cost-normalised frontier and finds no cost-equivalence crossover across the studied scales and hardware.

  • Tensor parallelism shards weights and KV memory across GPUs, while compression keeps one GPU and trades quality and dequantisation overhead for a smaller cache.
  • The two strategies are rarely priced against each other, leaving infrastructure owners without a direct cost comparison.
  • At fixed model, quality floor, and latency target, the study sweeps tensor-parallel degree and KV-compression settings using cost per million tokens against latency.
  • No cost-equivalence crossover appears at 7B or 70B models across the three tested GPU types.
  • The study reports a negative result that persists across two model scales and three hardware platforms rather than manufacturing the expected crossover.

B. KV-cache compression and eviction

KV-cache compression reduces memory through low-bit storage and eviction, but it must be distinguished from weight quantisation because weights determine some feasibility limits. The paper contributes a cost-normalised comparison that connects compression measurements with parallel-scaling results.

  • KV quantisation stores keys and values at low bit-width, while eviction drops lower-value tokens to reduce cache memory.
  • KV compression is distinct from weight quantisation because it shrinks the cache rather than the other large resident tensor.
  • The feasibility wall is set by weights, which KV compression cannot touch, although weight quantisation could move it.
  • Compression and parallel-scaling literatures report different metrics, motivating a direct comparison on cost per million tokens.
  • The study uses a profiled simulator to search deployment configurations that are too combinatorial for exhaustive measurement.

F. Our position

The paper treats a shared cost-normalised frontier as the missing link between tensor parallelism and KV compression. Its modelling choices expose communication, pricing, and quality assumptions while reporting a feasibility wall and complementary mechanisms instead of a crossover.

  • The study joins tensor-parallel and KV-compressed configurations on one cost-normalised frontier, with crossover analysis as its intended deliverable.
  • Cost per million tokens makes the strategies comparable by combining throughput, hourly price, and tensor-parallel device count.
  • Omitting the tensor-parallel degree from cost would make scale-out appear free and reverse the paper’s conclusions.
  • Tensor parallelism divides per-GPU weight and KV memory, adds all-reduce communication, and multiplies dollar cost by the degree.
  • Measured collective timings produce regime-dependent scaling that no single fitted communication coefficient can express.

D. Modelling KV compression

The KV-compression model reduces cache memory by changing element bit-width and resident-token count. Its latency treatment deliberately omits kernel-level compression effects and instead represents compression latency optimistically while bounding the missing penalty.

  • KV memory is modelled with bit-width b and eviction keep-ratio rkeep, alongside layers and per-worker KV heads.
  • The simulator replaces the fixed two-bytes-per-element assumption with b/8 and scales resident tokens by rkeep.
  • At b=16 and rkeep=1, the patched simulator reproduces the upstream simulator bit for bit.
  • Kernel-level latency consequences of compression are not modelled because the study lacks measurements of the competing byte-transfer and dequantisation effects.
  • The compression arm uses a best-case latency assumption and asks how large the omitted penalty would need to be to change the conclusion.

E. Feasibility

The study checks feasibility by requiring both weights and KV cache to fit device memory, then evaluates feasible tensor-parallel and compressed configurations on a latency–cost frontier. Infeasible configurations remain explicitly recorded rather than silently omitted or priced.

  • Feasibility rule: Feasibility requires weights and KV cache to fit within device memory with a μ=0.1 margin.The arithmetic feasibility check is tagged as exact (E).
  • Cost metric: Cost per million tokens combines per-GPU hourly price, simulated throughput, and the device-count multiplier for tensor-parallel configurations.The released dataset carries provenance flags for every metric row.
  • Evaluation procedure: The evaluation sweeps tensor-parallel and compression ladders, simulates feasible configurations, and tags results as simulated or exact.The workflow records out-of-memory cases explicitly before frontier construction.
  • Frontier analysis: The Pareto frontier retains feasible configurations that are not jointly dominated on latency and cost per million tokens.Tensor-parallel and compressed configurations are paired at matched memory relief, and the search checks for a cost sign change.

IV. EXPERIMENTAL SETUP

The experiments use Llama-2-7B and Llama-2-70B across A100, A40, and H100 settings with fixed synthetic workloads. Coverage is constrained by the simulator’s profiling support, the 4096-token context bound, and missing A40 all-reduce measurements.

  • Measurement provenance: Latency and throughput come from Vidur predictors calibrated on real A100, A40, and H100 profiling, not measurements collected by this study.The authors report 0.29 to 0.78% mean absolute percentage error on their runs and retain S/E provenance tags.
  • Models and hardware: The primary setup uses Llama-2-7B on an A100-80GB, with hardware comparisons on A40 and H100 and scale comparisons on Llama-2-70B.The 7B model uses multi-head attention with 32 KV heads, 32 layers, and 512 KiB of KV per token.
  • Serving configuration: The serving stack uses Sarathi-style chunked scheduling with block size 16 and a batch cap of 128.
  • Workloads: The realised workloads are synthetic fixed-length traces: W1 is interactive and unsaturated, while W2 is prefill-heavy under saturation.W1 uses 2048 prefill, 256 decode, and 64 requests; W2 uses 3840 prefill, 256 decode, and 384 requests at 30 QPS.
  • Coverage constraints: The simulator supports all four tensor-parallel degrees for Llama-2-7B, but context is bounded at 4096 tokens and A40 lacks all-reduce measurements for TP>1.Llama-3-8B was excluded because its profiling could not support the required tensor-parallel ladder.

V. RESULTS

Compression is cheaper than tensor parallelism at matched KV relief, while tensor parallelism improves latency and compression delivers much larger capacity per dollar. The cost ordering remains robust to plausible dequantisation overhead, though tensor parallelism can become cost-effective when feasibility constraints bind.

  • Cost-normalised frontier: 1.20×, 1.50×, and 1.89×: compression is cheaper than tensor parallelism at half, quarter, and eighth KV relief, respectively.The margin widens with deeper relief, so no cost-equivalence crossover appears within the swept range.
  • Latency and throughput: 4.5×: tensor parallelism cuts TTFT P99 from TP1 to TP8, but its throughput gains remain below break-even against device price.On W1, TP8 delivers 2.55× throughput for 8× spend; on W2, it delivers 4.45× throughput for 8× spend.
  • Capacity per dollar: 16.5×: compression raises concurrency capacity per GPU-dollar from 29.0 to 479.0, versus 21% from an eightfold GPU-spend increase.Tensor parallelism improves the figure from 29.0 to 35.1, making compression the stronger capacity-per-dollar lever.
  • Latency and throughput: 8% and 93%: INT4 increases TPOT P50 from 72.79 to 78.83 ms at 7B and from 116.76 to 224.81 ms at 70B.The paper attributes the latency penalty to batching contention from admitting more concurrent sequences.
  • Robustness: 16.8%, 33.5%, and 47.0%: compression would need to lose these throughput percentages at the three relief levels before tensor parallelism becomes cheaper.The reported published low-bit KV-kernel overheads are below the smallest threshold.

B. RQ2: where is the crossover, and how does it move?

At typical 7B deployment scale, an 80 GB device cannot exhaust its KV budget within the model’s context window. The decisive boundary is model size relative to device memory: KV compression cannot restore feasibility when weights alone exceed the device budget.

  • RQ2: crossover boundary: 29 concurrent requests: fp16/TP=1 on an 80 GB device remains feasible at the model’s maximum 4096-token context.OOM occurs only at 131,072 tokens, 32× beyond the context window, so the KV budget is not exhausted within that window.
  • RQ2: crossover boundary: Roughly 36B parameters: this is the feasibility boundary between CodeLlama-34B at 61.9 GB and Llama-2-70B at 127.5 GB on a 72 GB fp16 budget.The decision between compression and tensor parallelism therefore turns on which side of the model-size wall the deployment lies.
  • RQ2: crossover boundary: 127.5 GB versus 72 GB: Llama-2-70B and Llama-3-70B are infeasible at TP=1 because fp16 weights exceed usable device memory.They remain infeasible under INT4 with a 0.25 keep-ratio because KV compression does not shrink weights.

C. Above the threshold: Llama-2-70B

Above the memory threshold, Llama-2-70B requires tensor parallelism for feasibility, but compression remains cheaper at matched memory relief. Increasing degree can reduce cost when the deployment is memory-saturated, although all-reduce overhead makes high degrees unattractive.

  • Feasibility: TP=1 is infeasible for Llama-2-70B, so both configurations begin at TP=2 with memory relief measured relative to fp16/TP=2.The fp16/TP=2 baseline reaches 99–100% KV occupancy while holding roughly 13 concurrent requests.
  • Cost comparison: 1.10× and 1.56×: compression is cheaper than tensor parallelism at matched relief for INT8 and INT4, respectively.INT8/TP=2 costs $16.27 versus $17.95 for fp16/TP=4; INT4/TP=2 costs $15.48 versus $24.09 for fp16/TP=8.
  • Degree scaling: 3.38× throughput for 2× price: TP=2 to TP=4 on W1 reduces cost per token by 41%, while W2 improves by 14%.The gain occurs because TP=2 is capacity-starved rather than compute-starved; beyond TP=4, cost rises again.
  • Decision rule: The cheapest feasible degree is not always cheapest per token: when pmin is memory-saturated, one additional degree can be cheaper.This condition explains why TP=4 can outperform the minimum feasible degree above the threshold.
  • Degree scaling: 31.9% scaling efficiency at TP=8 on W1 means high degrees do not reach the cost frontier unless feasibility forces them.All-reduce overhead erodes the benefit of scaling as parallel degree increases.
  • Hardware comparison: Compression remains the cheapest arm on A100, A40, and H100, with H100 matched-relief margins of 1.25×, 1.48×, and 2.00×.On W2, cost per token is $1.96 for H100, $2.59 for A100, and $3.29 for A40.

VI. DISCUSSION

The paper concludes that tensor parallelism and KV compression are complementary rather than substitutable: parallelism makes weight-constrained deployments feasible, while compression provides cheaper concurrency headroom. The decision rule is to meet weight feasibility with minimal parallelism, adjust upward if saturated, then compress, subject to important simulation, quality, scope, and pricing limitations.

  • VI. DISCUSSION: Tensor parallelism is a feasibility mechanism for weights, whereas compression is a capacity mechanism for concurrency and does nothing for weights.Compression multiplies concurrency roughly 16× per dollar, while tensor parallelism is the only lever that improves latency.
  • VI. DISCUSSION: Choose pmin = ⌈2W/Musable⌉ for feasibility, add one degree if pmin is memory-saturated, then compress for concurrency.For Llama-2-70B, TP=4 restores feasibility at 128 requests and INT4+k25 increases capacity to 2054.
  • VI. DISCUSSION: Compression is cheaper at every matched relief level when both strategies are feasible; when compression fails, tensor parallelism is compulsory rather than economically superior.The pre-registered cost crossover was not supported.
  • VII. LIMITATIONS: The study has no measured GPU anchors: latency and throughput come from a simulator, with no held-out hardware error or planned fidelity ablation.Closed-form E-tagged results are unaffected by this limitation.
  • VII. LIMITATIONS: Compression results are upper bounds because no accuracy benchmark evaluated whether the quality floor permits each setting.The authors identify quality evaluation as the most consequential gap.
  • VII. LIMITATIONS: The approximately 36B threshold assumes fp16 weights; weight quantisation would move the boundary upward without removing the complementarity of the mechanisms.The evaluation uses one model family, 4096-token contexts, batch cap 128, synthetic workloads, and limited hardware coverage.
  • VII. LIMITATIONS: Cross-device ordering depends on relative prices, so the assumed $2.00, $1.00, and $3.50 GPU-hour rates should be replaced with the reader’s own contract.Uniform price changes preserve within-device ordering, but relative prices affect Fig. 11’s cross-device ordering.

VIII. CONCLUSION

Compression is cheaper than tensor parallelism at every matched level of memory relief, but model size relative to device memory determines whether compression is sufficient. Below the feasibility wall, extra GPUs are largely wasted spend; above it, tensor parallelism is compulsory because KV compression does not reduce weight memory.

  • Cost comparison: 1.20 to 2.00×: Compression is cheaper than tensor parallelism across matched memory relief levels and tested hardware regimes.The reported advantage is 1.20 to 1.89× below the feasibility threshold and 1.10 to 2.00× above it.
  • Robustness and next steps: 17 to 47%: Compression throughput would need to fall by this amount through dequantisation before the cost ordering reversed.The paper identifies measured low-bit KV decode kernels as the key assumption still requiring empirical anchors.
  • Decision boundary: Roughly 36B parameters: This is the decision boundary for an 80 GB device under fp16 weights.Below the wall, compression dominates; above it, tensor parallelism is compulsory because compression does not address weight memory.
  • Decision boundary: A 7B model on an 80 GB device cannot exhaust its KV budget within its context window.This makes additional GPUs largely wasted spend below the model-size wall.
  • Limitations and next steps: The study’s next steps are to close the quality axis, measure low-bit KV decode kernels, and relocate the wall under quantised weights.These additions target the quality floor, the bounded kernel assumption, and settings where weight quantisation changes feasibility.
Loading 2608.23962v1…