PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing AI Inference Across the Deployment Stack

Tejinder Singh, John Pflueger, Jeebak Mitra, Robert Lincourt, Mitchell Markow, Bhavesh A. Patel

arXiv:2609.10550v1cs.SEcs.LG

TL;DR

AI inference deployment lacks reliable cross-study comparisons because performance depends on interacting model, compiler, and serving choices measured under inconsistent conditions. This paper unifies those layers with analytical models and an evidence protocol, finding that cross-layer interactions—not any single optimization layer—govern deployment outcomes. It concludes with constraint-aware selection guidance while identifying important limits in the evidence base.

  • Problem

    Published inference benchmarks often report latency, throughput, and energy under incomparable hardware, software, batching, metric, and serving conditions, limiting rigorous deployment decisions.

  • Method

    The paper combines a three-layer deployment taxonomy, roofline and queuing models, multi-objective selection, and a protocol separating measured, derived, and analytical evidence.

  • Results

    Cross-layer interactions produce deployment phenomena that no single-layer analysis predicts, including a 125× framework latency spread, energy-ranking reversals, and quantization that can eliminate tensor parallelism.

  • Takeaways & Limitations

    Deployment choices should be evaluated jointly across model, compiler, and serving layers using constrained objectives and explicitly comparable evidence.

  • Takeaways & Limitations

    The evidence base lacks a controlled, same-condition multi-framework LLM serving benchmark with power measurement comparable to the edge study.

Abstract

from arXiv · show

AI deployment performance is shaped not by model architecture alone, but by interactions among compression, compiler transformations, and serving policies. Published benchmarks often report latency and throughput under incomparable conditions, limiting their use for deployment decisions. This paper presents a unified analytical treatment of inference optimization across the deployment stack. We introduce a three-layer taxonomy covering model-level techniques such as quantization, pruning, and distillation; compiler transformations such as graph fusion, layout optimization, and kernel autotuning; and system policies such as dynamic batching, admission control, and memory tiering. We formulate deployment as a constrained multi-objective optimization problem over accuracy, latency, throughput, memory footprint, and energy, and analyze a deployment-ranking functional with Pareto monotonicity and scale invariance. Roofline models show how memory-bandwidth hierarchies bound performance across precision regimes, while queuing models explain how service-time changes amplify response time under load. To improve comparability, we propose an evidence protocol that separates measured, derived, and analytical claims; limits numerical comparison to within-paper results; and requires reporting of hardware, software versions, batch semantics, and thermal state. We synthesize evidence from edge platforms, including Jetson AGX Orin and five inference frameworks; data center GPUs, including A100 and H100 with three LLM serving engines; and quantization studies across the Llama-3.1 family. The synthesis shows that deployment outcomes are governed by cross-layer interactions that no single-layer analysis can predict. We conclude with a constraint-aware selection procedure and open problems in compiler-serving co-optimization, cross-hardware performance prediction, and standardized energy reporting.

1 Introduction

Deployment performance depends on interactions across model compression, compiler optimization, and serving policies rather than model architecture alone. The paper frames deployment as a constrained, multi-objective problem grounded in hardware limits and comparable evidence.

  • Motivation: A 125× latency spread across frameworks for ResNet-152 on Jetson AGX Orin shows that deployment outcomes vary substantially even for the same model and hardware.Measured latency ranges from 2.3 ms with TensorRT to 285 ms with ONNX Runtime, while accuracy varies by up to 4.6 percentage points.
  • Deployment stack: The deployment stack combines model compression, compiler graph and kernel optimization, and system-level serving, with telemetry enabling iterative retuning across layers.The pipeline includes quantization, graph rewriting, kernel generation, batching, admission control, and memory management.
  • Hardware constraints: Roofline analysis distinguishes memory-bound from compute-bound kernels, showing why optimization must target the workload’s actual hardware bottleneck.On A100, L1/SMEM, L2, and HBM bandwidths are 19, 6, and 2 TB/s, while the FP16 Tensor Core ceiling is 312 TFLOP/s.
  • Multi-objective selection: Deployment selection is a constrained Pareto problem over accuracy and throughput rather than an unconstrained search across precision, compiler, batch size, and hardware choices.The design space contains dominated alternatives and a feasible region defined by minimum accuracy and throughput constraints.
  • Evidence comparability: Published latency, throughput, and power results are often incomparable because studies use different metric definitions, hardware, software, batching semantics, and thermal conditions.The paper therefore limits literal numerical comparison to aligned within-study results and distinguishes measured, derived, and analytical evidence.

2 Scope and Methodology

The methodology defines deployment analysis around explicit research questions and a design-space view of feasible configurations. It emphasizes that latency, throughput, and other reported metrics can refer to materially different quantities.

  • Measurement scope: Latency and throughput claims require careful interpretation because identical labels may describe different measurement quantities and serving conditions.The paper notes that studies can report fundamentally different forms of inference latency or throughput despite using the same terminology.
  • Deployment design space: The deployment design space represents precision, compiler, and batch-size configurations against accuracy and throughput constraints, separating feasible from dominated choices.Filled markers identify Pareto-optimal configurations, while dashed constraints define the feasible region.
  • Research questions: The scope is organized around five questions covering optimization levers, metric comparability, analytical models, cross-layer interactions, and platform-specific effects.These questions structure the paper’s evaluation of model, compiler, and system deployment decisions.

5. What open problems remain unresolved despite the current benchmark and systems literature?

The paper identifies unresolved problems in evidence quality, cross-system comparison, and deployment co-optimization. Its methodology addresses these through explicit evidence categories and conservative comparison rules, but the literature remains methodologically fragmented.

  • Inclusion criteria: The protocol excludes sources lacking essential deployment metadata, including batch size, precision format, measurement methodology, and hardware or software details.This conservative policy reflects sensitivity to thermal throttling, memory fragmentation, scheduling jitter, and version differences.
  • Evidence status: Deployment evidence is classified as measured, derived, or analytical so readers can distinguish instrumentation from computed quantities and model-based claims.Derived energy estimates depend on their component measurements, while analytical bounds depend on modeling assumptions.
  • Comparability: Cross-paper numerical comparison remains unreliable unless workloads, hardware, software, batch semantics, and metric definitions are explicitly aligned.The paper treats only within-paper numerical comparisons as literal and uses cross-paper results qualitatively otherwise.
  • Benchmark standards: Many benchmark studies predate or do not follow MLPerf scenarios, requiring their original results to be reported with explicit comparability limitations.MLPerf standardizes scenarios, quality targets, and reporting requirements, but the synthesized literature is not uniformly conformant.
  • Open methodological need: Prior technique-focused surveys leave an open need for pipeline-wide analysis that combines analytical models with transparent evidence comparability.The paper positions its contribution as complementary to surveys of compilers, quantization, pruning, and serving systems.

3 Deployment Taxonomy

Deployment optimization spans model, compiler, and system layers whose interactions determine whether theoretical improvements become practical gains. The section catalogs techniques at each layer and highlights hardware, compilation, and serving conditions that constrain their benefits.

  • Model Level Optimization: Quantization can deliver 2–4× throughput gains only when hardware exposes native INT8 or INT4 units; otherwise benefits mainly come from reduced memory footprint.Quantized tensors may be silently upcast to FP16 or FP32 on incompatible hardware.
  • Model Level Optimization: Pruning trades uncertain acceleration from high unstructured sparsity for more predictable execution from smaller dense graphs produced by structured pruning.Unstructured sparsity may not improve latency without efficient sparse GEMM kernels, whereas structured pruning runs on standard tensor cores but removes less compute.
  • Model Level Optimization: Distillation replaces the original model with a smaller student whose distinct computational graph changes operational intensity, memory access patterns, and compiler opportunities.Unlike pruning and quantization, distillation changes the architecture itself.
  • Compiler Level Optimization: Fusion reduces intermediate tensor materializations and HBM traffic, shifting memory-bound kernels toward higher operational intensity and potentially into the compute-bound regime.A fused attention kernel computes the operation in one SRAM pass instead of routing separate intermediates through HBM.
  • Compiler Level Optimization: Compiler autotuning searches among tile configurations and related kernel choices, but tuned configurations remain hardware specific and may transfer poorly across GPUs.A configuration tuned for an A100 may perform poorly on an H100 because of differences in memory, scheduling, and tensor-core microarchitecture.
  • System Level Optimization: Serving policies change the meaning of reported performance: static, dynamic, and continuous batching produce fundamentally different throughput quantities even with the same model and GPU.Continuous batching updates in-flight requests at decoding-step granularity, whereas static batching waits for a fixed request count.

4 Metrics and Measurement Semantics

The paper defines deployment metrics as a multidimensional vector whose meanings depend on measurement scope, serving semantics, and hardware conditions. It separates latency stages and treats energy, memory, and compilation cost as operationally relevant constraints.

  • The Deployment Metric Vector: Deployment candidates are characterized by accuracy, latency, throughput, memory, power, energy, and compilation cost across model, runtime, hardware, precision, batch, and serving choices.Accuracy is maximized; the other metrics generally represent costs or constraints.
  • Latency Semantics: Latency ranges from kernel execution to client-observed service time, including model execution, preprocessing, batching, queuing, postprocessing, and network delay.These variants can differ substantially, so comparisons must identify the reported latency definition.
  • Throughput Semantics: Throughput depends on batching and arrival conditions: offline saturation throughput is distinct from online request completion under realistic serving loads.Static batching approximates throughput as b/Tbatch, while continuous batching can improve utilization.
  • Power and Energy Scope: Energy per inference is derived as E = P · T, while power scope must distinguish accelerator, board, and total-platform measurements.Platform power is typically 1.5–2.5× accelerator-only power, and energy comparisons are invalid when measurement scopes differ.
  • Latency Decomposition: Request latency decomposes as Trequest(c) = Tqueue(c) + Tpre(c) + Twait(c) + Tmodel(c) + Tpost(c), separating queueing and batch-fill delays from execution and preprocessing.For autoregressive models, Tmodel consists of compute-bound prefill followed by sequential, memory-bound token decoding.
  • Operational Semantics: At ρ = 0.9, response time reaches 10Tmodel, showing that small service-time reductions have amplified effects near saturation.The paper also distinguishes online from offline throughput because SLA-constrained arrival rates can be much lower than saturation rates.

5 Analytical Framework

The analytical framework combines roofline bounds, queueing models, and constrained multi-objective selection to connect deployment choices with hardware limits and serving behavior. It ranks only feasible candidates and uses Pareto structure to resolve trade-offs.

  • Framework Overview: The framework combines roofline analysis, queueing theory, and multi-objective optimization to predict how deployment levers affect performance.These pillars connect hardware resource limits, load-dependent response time, and candidate selection.
  • 5.1 Roofline Grounded Inference Time Bounds: Roofline analysis identifies whether each kernel is compute bound or memory bound, determining whether reducing memory traffic or improving arithmetic efficiency can help.The bound is diagnostic rather than exact when pipelining and asynchronous execution create overlap.
  • 5.2 Queuing Amplification in Online Serving: The M/M/1 approximation assumes exponential service times, a single processor, and Poisson arrivals, whereas production inference violates each assumption.Variable-length generation, GPU concurrency, and bursty arrivals limit literal quantitative interpretation.
  • 5.2 Queuing Amplification in Online Serving: At ρ = 0.9, a 1 ms reduction in service time yields a 100 ms reduction in mean response time under the stated queueing approximation.More realistic M/G/1 analysis preserves the same near-saturation amplification while varying delay by service-time variability.
  • Constraint-Aware Selection: The selection procedure filters candidates by accuracy, latency, memory, and energy constraints before Pareto filtering and PEI ranking.Cheap accuracy and memory checks precede expensive latency and energy measurement.
  • Deployment Ranking: PEI is monotonic under Pareto dominance and invariant to positive unit conversions, but its ordering depends on application weights for non-dominating candidates.Thus, PEI cannot imply Pareto dominance and intentionally resolves trade-offs according to priorities.

6 Empirical Synthesis

The Jetson AGX Orin synthesis shows that framework performance varies sharply by model and objective even under controlled within-paper comparisons. Latency, accuracy, energy, and online serving behavior therefore produce different deployment rankings.

  • Framework Comparisons: ResNet-152 shows a 125× latency spread between TensorRT at 2.28 ms and ONNX Runtime at 285.5 ms on the same Jetson AGX Orin.The comparison attributes the gap to graph fusion and hardware-specific kernel generation versus minimal optimization.
  • Accuracy Variation Across Runtimes: ResNet-152 accuracy ranges from 76.6% top-1 under TensorRT to 72.0% under ONNX Runtime and JAX, a 4.6-point runtime gap.The paper links this variation to precision handling, graph transformation, and numerical accumulation order.
  • Energy Comparisons: TensorRT combines 28.3 W power with 0.065 J per inference for ResNet-152, whereas ONNX Runtime uses 14.2 W but 4.054 J because its latency is much higher.Energy is derived from power and latency rather than measured independently.
  • Framework Comparisons: Compiler headroom is model dependent: MobileNetV2 spans 8.4×, while Swin-T spans 1.9× across the evaluated frameworks.Figure 7 presents these framework spreads on a log scale.
  • Energy Comparisons: For Swin-T, TVM consumes 0.085 J versus TensorRT’s 0.108 J despite higher latency, because TVM draws 16.0 W versus 27.4 W.This makes TVM preferable under energy constraints and TensorRT preferable under latency constraints.
  • Online Serving Implications: At ρ = 0.8, TensorRT supports λ = 351 req/s with 11.4 ms mean response time, while ONNX Runtime supports 2.8 req/s with 1.43 s response time.The queueing model therefore predicts a much larger online-serving gap than the offline latency ratio alone.

6.6 Limitations of the Anchor Evidence

The evidence spans edge and data-center deployments, but its conclusions remain constrained by platform, model, and measurement scope. Results nonetheless show that cross-layer choices can reverse rankings and produce substantial gains.

  • Scope limitations: The Jetson study’s framework rankings may not transfer to data-center GPUs or LLMs, where hardware features and bottlenecks differ.The evidence covers edge vision models, while data-center GPUs and LLM serving involve different hardware capabilities, prefill/decode behavior, and KV-cache scaling.
  • Edge evidence: TVM uses 0.085 J versus TensorRT’s 0.108 J for Swin-T despite higher latency, because its power draw is 16.0 versus 27.4 W.This reverses the latency-only ranking and illustrates why energy-constrained deployments may select a different runtime.
  • Comparability boundary: These findings remain directional across studies because hardware, software versions, model families, and measurement methodologies differ.The paper therefore anchors claims to explicitly reported configurations and protocols where possible.
  • Data-center evidence: H100 FP8 reaches up to 4.6× higher maximum throughput and 4.4× faster first-token latency than A100 FP16 on Llama-2-13B.The reported conditions use TensorRT-LLM v0.5.0, TensorRT 9.1, SXM 80 GB, tensor parallelism 1, and batch sizes 1–64.
  • Data-center evidence: TensorRT-LLM leads H100 serving throughput at every concurrency level, while its advantage ranges from ∼8% at one request to ∼13% at 50 requests.At 100 requests, the gap compresses to ∼12% as queuing dynamics become more influential near saturation.
  • Cross-layer results: QServe achieves 1.2–1.4× higher throughput than TensorRT-LLM for Llama-3-8B and 2.4–3.5× for Qwen1.5-72B through quantization-serving co-design.It targets dequantization overhead with progressive quantization and recovers 4-bit KV-cache accuracy with SmoothAttention.
  • Quantization evidence: FP8 weight-and-activation quantization is effectively lossless, while INT8 degrades accuracy by 1–3% and INT4 weight-only quantization remains competitive across tasks.The evaluation spans the Llama-3.1 family and both academic benchmarks and open-ended generation assessments.
  • Quantization evidence: INT4 reduces Llama-3.1-70B weights from 140 GB to 35 GB, fitting the model on one 80 GB GPU instead of requiring two.Eliminating tensor-parallel communication can contribute more throughput than quantization alone.

7 Discussion and Limitations

The discussion argues that deployment optimization is irreducibly cross-layer, with evidence spanning compiler variation, energy trade-offs, serving dynamics, and quantization. It also identifies substantial limits in analytical assumptions, evidence coverage, hardware diversity, and benchmark durability.

  • 7.1 Strengths of the Cross-Layer Perspective: A 125× latency spread across frameworks for one model on fixed edge hardware shows that compiler choices can create two orders of magnitude of variation.The Swin-T energy reversal further shows that the optimal configuration changes with the binding metric.
  • 7.1 Strengths of the Cross-Layer Perspective: QServe’s throughput gains and narrowing framework gaps under high concurrency show that quantization, serving, and queuing effects interact across layers.The discussion connects QServe’s co-design result with the queuing model’s prediction near saturation.
  • 7.1 Strengths of the Cross-Layer Perspective: INT4 quantization can reduce a 70B model’s deployment from multiple GPUs to one by eliminating tensor-parallel communication.This is a system-level benefit triggered by a model-level optimization.
  • 7.2 Limitations of the Analytical Framework: The roofline model identifies bottleneck resources but can be conservative because kernels access multiple memory levels and overlap computation with memory operations.Effective bandwidth depends on kernel-specific data reuse patterns.
  • 7.2 Limitations of the Analytical Framework: The M/G/1 queuing model captures hockey-stick response-time growth near saturation but misestimates tail latency under continuous batching and admission control.It underestimates tail latency at moderate utilization and overestimates it at very high utilization where admission control intervenes.
  • 7.3 Structural Limitations of the Evidence: The paper lacks a controlled within-study comparison of LLM serving frameworks with the same rigor as the Jetson vision study.LLM prefill/decode asymmetry, KV-cache scaling, and continuous batching create qualitatively different bottlenecks.
  • 7.3 Structural Limitations of the Evidence: Most sources omit variance statistics, preventing assessment of whether framework differences are statistically significant or within measurement noise.The comparability policy restricts literal comparisons to within-study results but does not resolve the missing uncertainty information.
  • 7.3 Structural Limitations of the Evidence: The evidence base is concentrated on NVIDIA hardware and lacks a controlled, methodologically aligned comparison across three or more hardware platforms.AMD, Intel, Google TPU, and custom accelerators are underrepresented, limiting cross-platform generalization.

8 Design Guidance for Practitioners

Practitioners should treat deployment as a staged, constraint-aware selection process, measuring expensive metrics only after cheaper feasibility gates. The appropriate survivor depends on whether latency, throughput, memory, or energy is the binding objective.

  • 8.1 Staged Measurement Protocol: Fifty candidates can require only 15 expensive hardware profiles when memory or accuracy constraints eliminate 35 beforehand.The protocol orders free memory checks, cheap accuracy evaluation, and expensive latency, throughput, power, and energy measurements.
  • 8.2 Worked Example: Applying the Selection Algorithm: With equal weights, Swin-T/TVM wins on combined accuracy-energy, while latency-heavy weights select R-152/TRT after constraint filtering.In the Jetson AGX Orin example, 5 of 11 candidates fail accuracy, while the 6 survivors satisfy latency and energy constraints.
  • 8.3 Decision Tree for Common Scenarios: Latency-critical deployments should minimize T(c), throughput-critical systems should maximize Q(c) under the SLA, and memory-constrained systems should prioritize quantization.The guidance recommends TensorRT or AOT compilation for latency, continuous batching and PagedAttention for throughput, and INT4 weight-only quantization for memory.
  • 8.3 Decision Tree for Common Scenarios: Energy-constrained deployments should optimize E(c) = P(c) · T(c), because the energy-optimal framework may differ from the latency-optimal one.Lower-power runtimes can sacrifice latency while reducing total energy per inference.
  • 8.4 When to Re-evaluate: Re-profile after model, hardware, or workload changes because compiler optimizations, roofline regimes, and queuing conditions can shift.A configuration optimal at ρ = 0.5 may be suboptimal at ρ = 0.8, where the sensitivity factor 1/(1 −ρ)^2 is 6.25× larger.
  • 8.5 Anti-patterns: Benchmarking at the wrong batch size, ignoring energy, or quantizing without re-profiling can reverse rankings or miss cross-layer throughput gains.QServe achieves 2–3.5× higher throughput when quantization is co-designed with the serving runtime.

9 Open Research Problems

The paper identifies open problems where deployment decisions require jointly modeling compiler, hardware, and serving behavior. The main gaps are scalable co-optimization and reliable performance prediction across hardware generations.

  • 9.1 Joint Compiler-Serving Co-optimization: No general-purpose framework jointly optimizes compiler choices such as precision and fusion with serving choices such as batching and KV-cache allocation.The QServe result demonstrates 2–3.5× throughput gains from co-designing compilation and serving, but the combined search remains unresolved.
  • 9.1 Joint Compiler-Serving Co-optimization: Joint compiler-serving search is combinatorially large, motivating Bayesian optimization, reinforcement learning, or hierarchical decomposition.The cited search space reaches 10^5 combinations per operator.
  • 9.2 Cross-Hardware Performance Prediction: Roofline bounds do not reliably predict latency on new hardware because microarchitectural differences create nonlinear performance shifts.A model profiled on an A100 cannot reliably predict its latency on an H100 from roofline parameters alone.
  • 9.2 Cross-Hardware Performance Prediction: Cross-hardware latency prediction could reduce hardware-selection profiling from weeks to hours, but transfer learning requires expensive profiling datasets.Fine-tuning latency predictors across hardware generations is promising but data-intensive.

CROSS-LAYER RESEARCH PROBLEMS

The roadmap extends the framework toward joint optimization, reproducible measurement, and models tailored to production LLM serving. These directions address both comparability gaps and domain-specific performance phenomena.

  • Joint Compiler-Serving Co-optimization: Joint compiler-serving co-optimization must address precision, fusion, and batching across 10^5 combinations per operator.The roadmap identifies this as a cross-layer problem spanning compiler and system decisions.
  • Standardized Energy and Variance Reporting: Energy per inference and compilation cost remain underreported, limiting deployment comparisons especially for edge and sustainability-focused systems.The proposed standard requires sustained platform power, latency distributions, energy calculation, compilation time, and hardware/software metadata.
  • Standardized Energy and Variance Reporting: Standardized reporting would allow PEI ranking to be applied directly to published benchmark results.The required metadata include GPU, driver, CUDA, framework, precision, and batch size.
  • Reproducible Benchmark Artifacts: Reproducible benchmark artifacts should pin drivers, toolkits, frameworks, weights, and calibration data because software-version changes can invalidate published results.A container specification would turn the comparability policy into a preregistration requirement.
  • LLM-Specific Deployment Models: General roofline and queuing models miss LLM-specific effects including prefill/decode asymmetry, KV-cache growth, speculative decoding, and prefix caching.Proposed extensions include a two-phase roofline, heavy-tailed variable-service-time queuing, and memory-aware admission control.

10 Conclusion

The paper concludes that deployment performance is governed by interactions among model, compiler, and serving layers rather than any layer alone. It combines formal models, empirical evidence, and an evidence protocol to support constrained deployment decisions while identifying unresolved research gaps.

  • 10 Conclusion: Cross-layer interactions produce a 125× latency spread, energy-ranking reversals, and quantization-driven elimination of tensor parallelism.These examples show why single-layer analysis cannot predict deployment outcomes.
  • 10 Conclusion: Roofline, M/G/1 queuing, and PEI models respectively analyze bottlenecks, response-time amplification, and constrained multi-objective ranking.At ρ = 0.9, a 1 ms latency reduction yields a 100 ms response-time improvement.
  • 10 Conclusion: The comparability protocol separates measured, derived, and analytical evidence and restricts literal numerical comparisons to within-study results.It addresses differences in latency definitions, batch semantics, power scope, and software versions that can reverse framework rankings.
  • 10 Conclusion: Five open problems remain: compiler-serving co-optimization, cross-hardware prediction, standardized reporting, reproducible artifacts, and LLM-specific models.The paper frames deployment optimization as a constrained, multi-objective, cross-layer systems problem rather than a framework-ranking problem.
Loading 2609.10550v1…