PaperlayerView exact revision on arXiv

Source-linked AI summary

AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving

Tianhao Xu, Yiming Liu, Xianglong Lu, Yijia Zhao, Xuting Zhou, Aichen Feng, Yiyi Chen, Yi Shen, Qin Zhou, Xumeng Chen, Ilya Sherstyuk, Haorui Li, Rishi Thakkar, Ben Hamm, Yuanzhe Li, Xue Huang, Wenpeng Wu, Anish Shanbhag, Harry Kim, Chuan Chen, Junjie Lai

arXiv:2601.06288v1cs.LGcs.AIcs.DC

TL;DR

Production LLM inference tuning faces a costly, framework-specific configuration search across dynamic workloads and strict performance targets. AIConfigurator addresses this gap with operation-level performance modeling, calibrated measurements, and unified backend abstractions, evaluating configurations rapidly without GPU benchmarking. The system reports accurate predictions and substantial throughput gains in production-oriented case studies.

  • Problem

    LLM deployment tuning spans a combinatorial configuration space across parallelism, runtime flags, frameworks, and changing workloads, making manual optimization costly and often sub-optimal.

  • Method

    AIConfigurator decomposes inference into fundamental operations, combines measured hardware data with framework-specific performance modeling, and searches configurations through a unified backend abstraction.

  • Results

    6–12% MAPE for TPOT is reported across dense and MoE architectures, with production case studies showing 2× throughput improvements through automated disaggregated configuration discovery.

  • Takeaways & Limitations

    AIConfigurator evaluates thousands of configurations in seconds on CPU and supports optimized deployment across TensorRT-LLM, vLLM, SGLang, and NVIDIA Dynamo.

  • Takeaways & Limitations

    Performance modeling uses fixed user-provided output sequence length and sweeps predefined batch-size values, while the disaggregated mode requires valid total GPU counts and correction factors.

Abstract

from arXiv · show

Optimizing Large Language Model (LLM) inference in production systems is increasingly difficult due to dynamic workloads, stringent latency/throughput targets, and a rapidly expanding configuration space. This complexity spans not only distributed parallelism strategies (tensor/pipeline/expert) but also intricate framework-specific runtime parameters such as those concerning the enablement of CUDA graphs, available KV-cache memory fractions, and maximum token capacity, which drastically impact performance. The diversity of modern inference frameworks (e.g., TRT-LLM, vLLM, SGLang), each employing distinct kernels and execution policies, makes manual tuning both framework-specific and computationally prohibitive. We present AIConfigurator, a unified performance-modeling system that enables rapid, framework-agnostic inference configuration search without requiring GPU-based profiling. AIConfigurator combines (1) a methodology that decomposes inference into analytically modelable primitives - GEMM, attention, communication, and memory operations while capturing framework-specific scheduling dynamics; (2) a calibrated kernel-level performance database for these primitives across a wide range of hardware platforms and popular open-weights models (GPT-OSS, Qwen, DeepSeek, LLama, Mistral); and (3) an abstraction layer that automatically resolves optimal launch parameters for the target backend, seamlessly integrating into production-grade orchestration systems. Evaluation on production LLM serving workloads demonstrates that AIConfigurator identifies superior serving configurations that improve performance by up to 40% for dense models (e.g., Qwen3-32B) and 50% for MoE architectures (e.g., DeepSeek-V3), while completing searches within 30 seconds on average. Enabling the rapid exploration of vast design spaces - from cluster topology down to engine specific flags.

1 Introduction

LLM inference configuration is difficult to optimize because workloads, hardware, serving architectures, and framework-specific runtime settings create a large, costly search space. AIConfigurator addresses this challenge with data-driven operation-level modeling, rapid configuration search, and production-framework integration.

  • Challenges: Production LLM tuning spans parallelism, batching, quantization, serving architecture, and framework-specific runtime flags.These interacting choices make manual benchmarking expensive and often leave deployments sub-optimal.
  • Challenges: Black-box optimizers and automated serving frameworks still require substantial GPU hours for each scenario.
  • Approach: AIConfigurator decomposes inference into kernels such as GEMM, attention, and communication, then interpolates real system measurements for high-fidelity estimates.The toolkit targets NVIDIA Ampere, Ada, Hopper, and Blackwell platforms.
  • Contributions: AIConfigurator navigates complex configuration spaces in seconds and integrates with vLLM, SGLang, TRTLLM, and NVIDIA Dynamo.The system produces actionable, production-ready configuration recommendations.
  • Evaluation: The evaluation benchmarks AIConfigurator against ground-truth silicon data.

2 Background

LLM serving combines scheduling, model parallelism, and configuration tuning to balance prefill and decode workloads. Disaggregated serving enables independent scaling but introduces transfer overhead, so the best architecture depends on workload, bandwidth, and deployment scale.

  • Serving architecture: Scheduling methods address the distinct computational profiles of prefill and decode to improve hardware utilization.Relevant techniques include continuous batching, PagedAttention, chunked prefills, and disaggregated serving.
  • Serving architecture: Disaggregated serving separates prefill and decode across GPU pools, enabling independent scaling while adding KV-cache transfer overhead.
  • Serving architecture: Aggregated serving with chunked prefills can outperform disaggregation for smaller deployments when transfer overhead outweighs its benefits.The preferred architecture depends on workload mix, interconnect bandwidth, and cluster scale.

3 Motivation

Existing heuristics and static recipes do not reliably capture nonlinear hardware interactions or changing production conditions. AIConfigurator motivates algorithmic, SLA-aware search across frameworks, parallelism, batching, and serving architectures.

  • Configuration search: Traditional heuristics can miss nonlinear interactions between computation and network bandwidth.Automated search has been reported to outperform manual tuning by more than 2× in cost-efficiency.
  • Framework heterogeneity: Frameworks such as vLLM, SGLang, TensorRT-LLM, and NVIDIA Dynamo expose distinct performance characteristics and runtime performance cliffs.Generic models do not effectively capture these framework-specific behaviors.
  • Configuration search: Static benchmark tables and curated recipes are insufficient for dynamic production environments.AIConfigurator instead searches for SLA-compliant configurations across parallelism, batch sizes, and serving architectures.

4 Design and Implementation

AIConfigurator combines measured operator performance with framework-specific simulation to search serving configurations, analyze Pareto-optimal choices, and generate backend-compatible launch files. Its workflow models multiple serving modes and selects configurations that maximize throughput under latency and speed constraints.

  • Architecture: AIConfigurator composes real hardware measurements of fundamental operations into end-to-end performance estimates.The system avoids both purely theoretical models and exhaustive benchmarking.
  • Architecture: A unified backend abstraction shares operation modeling while implementing framework-specific memory estimation, serving simulation, and constrained optimization.
  • Pareto analysis: Figure 1 compares throughput-versus-speed Pareto frontiers for Qwen3-235B on 64 H200 GPUs under a TTFT constraint of 1000ms.The horizontal axis is generation speed per user request and the vertical axis is system throughput per GPU.
  • Pareto analysis: 823 tokens/s/GPU is achieved by the best disaggregated configuration versus 564 tokens/s/GPU for aggregated serving under the same speed constraint.The disaggregated configuration is approximately 53% higher in this example.
  • Optimization objective: AIConfigurator identifies configurations that maximize throughput while satisfying SLA targets such as TTFT below 1 second and generation speed above 20 tokens/s/user.
  • Serving modes: The system models static, aggregated, and disaggregated serving modes.Aggregated mode mixes prefill and decode steps, whereas disaggregated mode uses separate prefill and decode GPU pools.
  • Workflow: The workflow constructs candidate configurations from workload and SLA requirements, then estimates metrics using iteration-level models and operator data.
  • Deployment: The Pareto analyzer ranks valid configurations, and the generator converts recommendations into compatible launch files with optimized serving flags.Supported flags include CUDA graphs, KV-cache memory fraction, and chunked context settings.

4.2 Performance Modeling

AIConfigurator estimates serving performance from request lifecycle phases and explicitly models static, aggregated, and disaggregated serving modes. It derives TTFT and TPOT using phase-specific latency estimates, scheduling behavior, and SLA-constrained configuration search.

  • Metric Estimation: TTFT measures first-token latency, while TPOT averages output-token latency over the generated sequence.Both metrics are measured in milliseconds, and output sequence length is supplied by the workload descriptor.
  • Request Lifecycle: LLM requests pass through compute-intensive prefill and memory-intensive autoregressive decode phases.Prefill processes the prompt and builds the KV cache; decode generates remaining tokens while reusing cached key-value pairs.
  • Static Mode: Static serving processes a fixed batch sequentially, with TTFT tied to prefill latency and TPOT averaged over subsequent decode steps.A stride-based optimization interpolates generation costs across token intervals instead of querying every step.
  • Aggregated Mode: Aggregated serving mixes prefill and decode requests, using a mixed phase and a generation-only phase to model continuous batching.When prefill is heavy, rate matching throttles concurrent decode streams; TTFT uses a corrected mixed-phase estimate, while TPOT combines both phases.
  • Disaggregated Mode: Disaggregated serving searches valid prefill/decode worker compositions by rate matching under TTFT and TPOT constraints.For a selected (x)P(y)D configuration, TTFT comes from prefill latency including transfer overhead, and TPOT comes from decode latency.

4.3 Iteration-level Modeling

AIConfigurator models each inference iteration as repeated execution of a fixed operator sequence, augmented by communication operators introduced by parallelism. This decomposition supports latency prediction by aggregating constituent operator performance.

  • Iteration-level Modeling: Transformer-layer repetition makes an inference step a fixed operator sequence whose compute operators scale with sharding.Parallel strategies add well-defined communication operators at fixed execution positions.
  • MoE Operator Composition: An MoE inference step includes embedding, GEMM, attention, and MoE operators repeated according to the model structure.Expert parallelism reduces the MoE operator size and adds backend-dependent communication operators.
  • Iteration-level Modeling: Inference iteration latency is predicted by aggregating performance profiles of constituent operators.The model relies on latency data for prefill-only, decode-only, and mixed steps across the serving algorithms.

4.4 Operator Database

AIConfigurator combines framework-specific GPU performance databases with interpolation and analytical bounds to estimate operator latency. Its MoE modeling also reproduces production-like expert-load imbalance through controlled token assignments.

  • Database Construction: The performance database supports TensorRT-LLM, vLLM, and SGLang using offline profiling on actual GPU hardware.Framework backends share operation modeling while implementing framework-specific memory estimation, serving simulation, and optimization logic.
  • Database Coverage: The database covers GEMM, attention, communication, and MoE dispatch/combine operations across relevant dimensions, precisions, attention types, message sizes, and GPU counts.Attention coverage includes compute-bound context and memory-bound generation modes, including MHA, GQA, and MLA.
  • Data Collection: Data collection combines exhaustive profiling, interpolation for intermediate configurations, and Speed-of-Light roofline bounds for unprofiled operators.Exhaustive sweeps take approximately 30 GPU-hours per platform-framework pair.
  • MoE Load Modeling: AIConfigurator models MoE imbalance with power-law expert loads rather than assuming uniform token distribution.The method samples expert weights, normalizes them into token counts, and uses α to control skew; α ≈ 1.2 yields heavy-tailed loads matching Qwen3-235B observations.
  • MoE Load Modeling: Synthetic router assignments deterministically route exactly N_i tokens to expert i, enabling controlled measurement of workload-dependent tail latency.By bypassing stochastic routing, the hardware executes the precise token-distribution shape generated by the load model.

5 Evaluation

AIConfigurator is evaluated for prediction fidelity, search efficiency, and production configuration optimization across aggregated and disaggregated serving. It provides accurate predictions across frameworks and identifies higher-throughput configurations under SLA constraints while avoiding exhaustive GPU benchmarking.

  • Evaluation scope: AIConfigurator evaluates prediction fidelity, search efficiency, and production performance across aggregated and disaggregated serving scenarios.The evaluation spans TensorRT-LLM and vLLM, dense and MoE models, and production deployment configurations.
  • Aggregated serving fidelity: 7.8% overall MAPE is achieved for TPOT prediction across TensorRT-LLM and vLLM configurations.Errors are 8.2% for Qwen3-32B-TRTLLM, 6.8% for Qwen3-235B-MoE-TRTLLM, and 11.9% for Qwen3-32B-VLLM, with strong correlations.
  • Aggregated serving fidelity: 16.9–22.1% MAPE is reported for TTFT prediction across the evaluated TensorRT-LLM and vLLM workloads.The reported correlations are r=0.89 for Qwen3-32B-TRTLLM, r=0.66 for Qwen3-235B-MoE-TRTLLM, and r=0.95 for Qwen3-32B-VLLM; TTFT values above 1000ms are excluded as outliers.
  • Disaggregated serving fidelity: 13.19% throughput MAPE and 3.35% generation-speed MAPE are achieved in the 25–50 tokens/s/user interactive region for disaggregated DeepSeek V3 serving.Across all configurations, the corresponding MAPEs are 25.49% for system throughput and 14.94% for generation speed.
  • Search efficiency: 427,000× faster search is achieved for Qwen3-235B, requiring 0.8s on CPU versus 99.5 GPU-hours for equivalent benchmarking.AIConfigurator’s median per-configuration simulation time remains approximately 1.5ms regardless of model size.
  • Production case study: 101.6% higher throughput is achieved by the disaggregated Qwen3-32B configuration compared with the aggregated baseline while satisfying all SLA constraints.The disaggregated setup reaches 648.3 tokens/s/GPU versus 321.5 tokens/s/GPU for the aggregated baseline; its projected frontier deviates by at most 11.2% in generation speed and 17.4% in system throughput.

6 Related Work

Related work covers dynamic LLM serving, model parallelism, disaggregated architectures, and performance simulation. AIConfigurator differs from analytical simulators and static recipes by composing measured operation timings while modeling framework-specific behavior.

  • LLM inference systems: Modern LLM serving combines dynamic scheduling, model parallelism, and disaggregated prefill/decode architectures with framework-specific performance trade-offs.The surveyed systems include vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo, and parallelism strategies such as TP, PP, and EP.
  • Performance simulation: AIConfigurator composes measured execution times to capture implementation-specific overheads that analytical models may miss.Its unified backend abstraction supports framework-specific memory estimation, serving simulation, and constraint-based optimization.

7 Conclusion

AIConfigurator is a data-driven toolkit for optimizing LLM inference configurations across multiple serving frameworks. Its evaluation reports accurate predictions, rapid CPU-based search, and substantial throughput gains from automated disaggregated configuration discovery.

  • Contribution: AIConfigurator decomposes inference into fundamental operations and measures their latencies on target hardware to model framework-specific overheads.The toolkit targets TensorRT-LLM, vLLM, and SGLang.
  • Search efficiency: Thousands of configurations can be evaluated in seconds on CPU, eliminating expensive GPU benchmarking campaigns.The conclusion emphasizes rapid configuration exploration without exhaustive GPU measurement.
  • Results: 6–12% MAPE for TPOT and 2× throughput improvements are reported across dense and MoE production workloads.The throughput improvements arise from automated disaggregated configuration discovery in the reported case studies.
Loading 2601.06288v1…