PaperlayerView exact revision on arXiv

Source-linked AI summary

The Inference Engineering Pareto Atlas: Which Optimizations Dominate the Cost, Quality, and Latency Frontier?

Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly

arXiv:2609.17863v1cs.AI

TL;DR

LLM inference optimizations are difficult to compare across incompatible evaluation settings, leaving deployment choices unclear. This paper uses measured anchors and a calibrated simulator to build a cost-quality-latency Pareto atlas. FP8 weights are a quality-preserving option among the regime winners, while measured quality can exclude fast configurations.

  • Problem

    Speedups reported across different models, GPUs, prompts, and quality metrics are difficult to compare or combine for deployment decisions.

    Source: §I · p. 1

  • Method

    The paper measures Qwen2.5-7B-Instruct anchor configurations, calibrates a profiled simulator, and builds a cost-quality-latency atlas for optimizations and combinations.

    Source: §I · p. 1

  • Results

    Three of four regime winners use FP8 weight quantization, while the naive FP8 KV cache scores 0/200 on both GSM8K accuracy metrics.

    Source: F. The measured quality axis · p. 6, §VII · p. 7

  • Takeaways & Limitations

    The best configuration depends on the binding constraint and GPU, and quality must be considered alongside speed when choosing among operating points.

    Source: C. The atlas · p. 5, F. The measured quality axis · p. 6, §VII · p. 7

    The atlas is bounded by the covered optimizations, models, GPUs, workloads, and price assumptions; sparse attention is modeled rather than measured.

    Source: §VI · pp. 6–7

Abstract

from arXiv · show

LLM inference optimizations report speedups on different models, GPUs, prompts, and quality metrics, making them hard to compare or combine. We build a cost, quality, and latency Pareto atlas to identify the best configurations for different deployment constraints. Since exhaustive testing is impractical, we measure 54 configurations of Qwen2.5-7B-Instruct running on vLLM 0.12 across L4, A100, and H100 GPUs and use these anchors to calibrate a simulator. It reproduces measurements at anchored batch sizes, with cross campaign drift below 1.5 percent. A separate quality evaluation tests FP16, AWQ 4bit, FP8 weights, and FP8 KV cache on 200 GSM8K questions with five examples per prompt. Sparse attention is evaluated only in simulation. On the calibrated grid, 18 of 36 configurations reach the Pareto frontier. Combined methods reach it more often than individual methods, with 9 of 15 combinations versus 9 of 21 single methods. Quality testing changes the winners. AWQ 4bit reduces per token latency to 0.34 times baseline on L4 but loses 5.9 percent of strict GSM8K accuracy, narrowly missing the 95 percent quality floor within sampling uncertainty. Flexible answer extraction matches FP16 accuracy, suggesting the loss comes from formatting rather than arithmetic. FP8 weights retain 99.4 percent of baseline accuracy at 0.61 to 0.65 times baseline latency across all three GPUs and appear in three of four regime winners. A naive FP8 KV cache maintains normal throughput but answers none of the 200 questions correctly, showing why speed alone is insufficient. Under two prompt designs, n gram speculative decoding measures at 0.90 to 0.98 times baseline and adds no benefit on this stack. The best choice depends on the constraint and GPU: H100 wins for tight latency, while A100 wins for throughput and low cost at 0.106 dollars per million tokens.

I. INTRODUCTION

Inference speedups come from incompatible models, hardware, prompts, and quality metrics, leaving practitioners without a common basis for choosing methods under deployment constraints. The paper builds a cost-quality-latency Pareto atlas, using measured anchors and a calibrated simulator to compare methods and combinations across regimes.

I. INTRODUCTION

  • Different models, GPUs, prompts, and quality metrics make published speedups difficult to compare, combine, or trust for a given deployment.This leaves engineers unable to choose a winner for fixed latency, cost, or quality constraints when method interactions are unknown.Source: §I · p. 1
  • The paper measures anchor configurations, calibrates a profiled simulator, and uses it to map dominant configurations under tight-latency, high-throughput, and low-cost constraints.The measurements cover Qwen2.5-7B-Instruct on one serving workload across an L4, A100, and H100.Source: §I · pp. 1–2
  • The atlas places major inference optimizations and promising combinations on common axes to identify the dominant configuration in each deployment regime.Its measure-then-simulate method calibrates against real runs so the broader configuration space can be covered from measured anchors.Source: §I · pp. 1–2, Figure 1 · p. 2
  • The study presents practitioner rules for which optimizations and combinations recur on the frontier and how choices shift with constraints and GPU.The atlas reports a regime map rather than treating one speedup number as decisive.Source: §I · p. 2, Figure 2 · p. 2

II. BACKGROUND AND RELATED WORK

Inference optimizations target different costs, while existing benchmarks and economics studies do not jointly map the full optimization stack and hardware onto a cost-quality-latency frontier.

A. The optimization stack

  • Quantization and KV-cache methods target memory and bandwidth, speculation targets decode steps, batching targets utilization, and sparse attention targets attention compute.The stack also includes prefill-decode disaggregation, which separates the two phases; each method exposes a knob that shifts a different axis.Source: A. The optimization stack · p. 2

B. Benchmarks and economics

  • Existing serving, speculation, and inference-economics benchmarks do not jointly place the full optimization stack on a cost-quality-latency frontier with an explicit dominance map.The atlas adds optimization and hardware axes to prior work on economics, cost-quality frontiers, and local-inference benchmarking.Source: B. Benchmarks and economics · p. 2

C. Simulation for large configuration spaces

Because exhaustive measurement of the combinatorial configuration space is infeasible, the paper adopts measure-then-simulate and treats comparability and dominance as its deliverable.

C. Simulation for large configuration spaces

  • The paper measures real-hardware anchors, calibrates a profiled simulator, and uses it to fill the frontier across the configuration space.This approach builds on Vidur [2], which estimates LLM-inference latency and throughput within a few percent and searches hundreds of deployment configurations cheaply.Source: C. Simulation for large configuration… · p. 3

D. Pareto dominance

  • A configuration is dominated when another is at least as good on every axis and strictly better on one; the frontier contains the non-dominated choices.The binding constraint determines which frontier configuration is optimal.Source: D. Pareto dominance · p. 3

E. Our position

  • The paper defines its deliverable as a common protocol and axes, calibrated simulator, and reported frontier and dominance map for practitioners choosing optimizations.The configuration space includes every optimization and combination placed on the common axes.Source: E. Our position · p. 3

III. METHODOLOGY

The methodology evaluates configurations on quality, latency, and cost axes, then measures anchors, simulates the broader configuration space, and computes regime-specific dominance.

A. Axes and metrics

  • Each configuration is placed on quality, latency, and cost axes, with throughput and GPU memory recorded as supporting metrics.Quality is task accuracy or agreement with the full-precision model; latency includes TTFT and TPOT at P50 and P99, while cost is dollars per million tokens derived from throughput and GPU hourly price.Source: A. Axes and metrics · p. 3
  • The atlas measures anchor configurations on real hardware, calibrates a profiled simulator, simulates the full space, and computes the Pareto frontier and dominant configuration for each regime.The regimes are tight-latency, high-throughput, and low-cost.Source: A. Axes and metrics · p. 3, Figure 3 · p. 3

B. Configurations and combinations

  • A configuration specifies optimization settings for a model and GPU, while combinations stack compatible methods.Settings can include quantization bit-width, KV keep-ratio or bit-width, speculation draft, batching policy, or sparse-attention budget.Source: B. Configurations and combinations · p. 3

C. Measure then simulate

The atlas is built by measuring real-hardware anchors, calibrating a profiled simulator, and using it to map deployment regimes and dominant configurations. Cost per million tokens combines GPU hourly price with measured throughput, while simulated points carry calibrated error bars.

C. Measure then simulate

  • Measured anchors calibrate a profiled simulator that fills the remaining configuration space.The simulator’s latency and throughput predictions are matched to real runs within a small tolerance.Source: C. Measure then simulate · p. 3

D. Reading the atlas

  • Atlas rows represent deployment regimes, while columns represent axes or workloads; each cell names the dominant configuration.The tested expectations include recurring combinations, constraint-dependent winners, and GPU-driven frontier shifts.Source: D. Reading the atlas · p. 3

E. Cost accounting

  • Cost per million tokens depends on GPU hourly price and measured throughput, and simulated points carry calibrated error bars.Measured anchors are shown without simulator error bars, and frontiers distinguish which points are anchors.Source: E. Cost accounting · p. 3

A. Models, workloads, hardware, and metrics

The measurement campaign uses Qwen2.5-7B-Instruct on one serving workload across three GPUs, with GSM8K as the measured quality workload. It anchors selected optimizations on real hardware and simulates the remaining combinations.

A. Models, workloads, hardware, and metrics

  • The campaign measures Qwen2.5-7B-Instruct on a 512-token prompt and 128-token output across L4, A100 80GB PCIe, and H100 PCIe GPUs.All three GPUs use vLLM 0.12; GSM8K is the only measured protocol workload.Source: A. Models, workloads, hardware, and… · pp. 3–4, Table I · p. 4
  • The atlas pipeline measures anchors, calibrates a profiled simulator, fills the configuration space, and labels dominant configurations on the Pareto frontier.The anchor measures include quality, TTFT, TPOT, throughput, and cost.Source: Figure 4 · p. 4
  • The metric panel includes quality, TTFT, TPOT at P50 and P99, throughput, GPU memory, and cost per million tokens.Reported values specify the model, GPU, batch size, and workload.Source: A. Models, workloads, hardware, and… · p. 4

B. Optimizations and protocol

  • The campaigns anchor AWQ-4bit and online FP8 weights, FP8 KV cache, and n-gram prompt-lookup; sparse attention is simulated only.Draft-model speculation is unimplemented in the serving engine used, and the target model has no EAGLE head.Source: B. Optimizations and protocol · p. 4

V. RESULTS

On the calibrated grid, 18 of 36 configurations are non-dominated, but the zero-quality FP8-KV-cache family is excluded from deployment regimes by quality guards. The cost frontier places an FP8, batching, and sparse stack at the A100 low-cost end, while cheaper AWQ falls below the measured quality floor.

V. RESULTS

  • Latency, throughput, and cost results use 54 measured anchor configurations, a calibrated simulator, and a measured GSM8K quality arm.Anchors were cross-campaign consistent within 1.5%; total spend was $18.Source: §V · pp. 4–5

A. The quality-latency frontier

  • 18 of 36 configurations are non-dominated on the calibrated quality-latency grid, while the zero-quality FP8-KV-cache family is excluded from deployment regimes.The raw frontier and the atlas differ because the atlas applies quality guards.Source: A. The quality-latency frontier · p. 5

B. The cost-quality frontier

  • $0.106/Mtok is the A100 low-cost anchor for the FP8 + batching + sparse stack; cheaper AWQ falls below the measured quality floor.The KV-cache arm collapses to zero quality, while speculation measures at parity with baseline under both prompt designs.Source: B. The cost-quality frontier · p. 5

C. The atlas

The dominant configuration changes with the deployment constraint and GPU: H100 takes tight latency, while A100 takes throughput, cost, and the quality floor. FP8 weights appear in three of four regime winners, and their TPOT ratio stays nearly flat across GPUs.

C. The atlas

  • H100 takes tight latency, while A100 takes throughput, cost, and the quality floor; FP8 weights appear in three of four regime winners.The atlas identifies the dominant configuration for each deployment regime.Source: C. The atlas · p. 5

D. How the frontier moves with hardware

  • AWQ-4bit TPOT ratios are 0.53× on H100, 0.49× on A100, and 0.34× on L4, while FP8 weights hold at 0.61 to 0.65× across GPUs.The paper identifies FP8 as the portable default across all three GPUs.Source: D. How the frontier moves with hardware · p. 5

C. The atlas

  • Stacks reach the frontier more often than single methods: 9 of 15 combinations versus 9 of 21 single methods.The comparison is on the calibrated grid.Source: Figure 8 · p. 6

E. Which optimizations recur, and which combinations win

Combined methods more often reach the frontier than single methods, but quality screening changes which configurations remain viable. The calibrated atlas reports winners under three constraints and checks simulator fidelity against measured anchors.

E. Which optimizations recur, and which combinations win

  • 9 of 15 stacked configurations reach the frontier versus 9 of 21 single-method configurations (60% vs 43%), but quality testing removes KV-bearing stacks from every regime.Quantization appears in 13 of 18 frontier members; speculation and sparsity appear in 6 each, KV compression in 4, and batching in 3.Source: E. Which optimizations recur, and… · p. 6

F. The measured quality axis

  • FP8 weights score 0.760 strict / 0.805 flexible, effectively lossless; AWQ-4bit scores 0.720 / 0.810, with its strict drop attributed to format discipline rather than arithmetic.Flexible extraction for AWQ ties baseline accuracy.Source: F. The measured quality axis · p. 6
  • The naive FP8 KV cache scores 0/200 on both metrics despite normal measured throughput, so throughput-only selection would retain a configuration that answers every question wrong.Joint cost-quality-latency accounting removes it.Source: F. The measured quality axis · p. 6

G. Picking under a constraint, and simulator fidelity

VI. DISCUSSION AND LIMITATIONS

The atlas identifies configuration winners under deployment constraints, while its conclusions remain bounded by the measured quality arm and modeled or unanchored methods. The authors present the protocol—not fixed coordinates—as the durable deliverable.

VI. DISCUSSION AND LIMITATIONS

  • Absolute positions shift with kernel maturity and prices; the atlas is bounded by its covered optimizations, models, GPUs, workloads, and stated price assumptions.Decisions near a frontier edge should be confirmed by direct measurement.Source: §VI · p. 6
  • The AWQ-versus-quality-floor verdict is a boundary case inside the reported error bar, so the evidence does not establish a clear separation between AWQ and the quality floor.Source: §VI · p. 7
  • N-gram speculation measured at parity (0.90 to 0.98×) under two prompt designs; draft-model speculation could not be anchored on this vLLM stack.The V1 engine raises NotImplementedError for draft-model speculation, and no EAGLE head exists for Qwen2.5-7B-Instruct.Source: §VI · p. 7
  • Sparse attention is simulated only; the FP8-KV anchor uses naive, uncalibrated scales, leaving scale-calibrated KV and eviction methods for future anchors.Sparse attention lacks a stock-vLLM implementation.Source: §VI · p. 7
  • Calibration is interpolative at anchored batch sizes, so 1.5% cross-campaign drift—not zero on-anchor residual—bounds generalization.The reported cost assumptions use July-2026 RunPod rates, which shift with the market.Source: §VI · p. 7

VII. CONCLUSION

  • FP8 weight quantization appears in three of four regime winners; AWQ falls at the measured quality floor, naive FP8 KV is 0/200 correct, and implementable speculation adds nothing.A 50% A100 price shock moves the cost and quality-floor winners to the L4.Source: §VII · p. 7
  • Re-running the atlas after price or kernel changes is the durable deliverable: anchors, calibration, frontier, and atlas.The atlas lets practitioners read off a choice for a budget, quality floor, or latency target.Source: §VII · p. 7

Related papers

Loading 2609.17863v1…