PaperlayerView exact revision on arXiv

Source-linked AI summary

InferOpt: Constrained Multi-Objective Search for LLM Inference Configurations

Qi Chen, Yingying Cheng, Zhaoyi Sun, Li Zhou, Fan Zhang, Jie Sun

arXiv:2610.04473v1cs.AI

TL;DR

Layer-wise LLM inference settings create high-dimensional configuration spaces where mechanism-specific heuristics provide narrow exploration and a single operating point. InferOpt applies constrained multi-objective search using a frozen proxy, resource-aware budget checks, and full-dataset validation. It outperforms three search baselines in both KV and MoE settings, with searched configurations improving quality-efficiency trade-offs against the stated references.

  • Problem

    Mechanism-specific heuristics explore layer-wise inference configurations narrowly and return a single operating point rather than a quality-efficient Pareto frontier for varying deployment budgets.

    Source: §1 · p. 1

  • Method

    INFEROPT uses constrained multi-objective search with a frozen proxy set, deterministic resource-cost checks before evaluation, and full-dataset validation of Pareto representatives.

    Source: §1 · pp. 1–2, §2 · p. 2

  • Results

    INFEROPT outperforms Random Search, NSGA-II, and MOTPE in both settings; searched KV allocation improves quality by 3.21–6.13 points over uniform allocation, while the searched MoE schedule reduces routed work by 43.0% within 0.59 points of default.

    Source: §4 · p. 4

  • Takeaways & Limitations

    A shared budget-aware search pipeline supports continuous KV-retention and discrete MoE-routing spaces without mechanism-specific redesign.

    Source: §4 · p. 4

    Resource cost is an architecture-level proxy for retained KV blocks or activated routed experts, not measured latency, memory, or energy.

    Source: §2 · p. 2

Abstract

from arXiv · show

Serving an LLM means setting dozens of inference-time knobs, from per-layer KV retention to per-layer expert counts. Practice sets them with mechanism-specific heuristics that return a single operating point and do not scale to layer-wise search spaces. We recast inference configuration as constrained multi-objective black-box optimization and build InferOpt, a reusable search framework that requires only variable bounds, a deterministic resource cost, and an evaluation hook. InferOpt searches on a frozen sampled proxy set, rejects over-budget candidates before any model call, tightens the budget adaptively, and re-validates Pareto representatives on full-scale dataset. One pipeline covers a 28-dimensional continuous KV space (Qwen2.5-7B) and a 26-dimensional discrete MoE space (DeepSeek-V2-Lite). On KV, post-prefill pruning cuts the 16K cache by 64.4% and TPOT by 22.9--48.5%, and the searched layer-wise budget by InferOpt beats a matched uniform budget by 7.3% and 14.0% of the Full KV reference points. On MoE, a searched top-k schedule by InferOpt removes 43.0% of routed token--expert pairs while staying within 0.59 points of the default, closer than the matched-budget baselines. Against Random Search, NSGA-II, and MOTPE, InferOpt leads on both spaces, taking the best proxy hypervolume and the lowest retention on KV and staying closest to the uncompressed reference at the lowest experts on MoE.

1 Introduction

Layer-wise inference configurations create high-dimensional search spaces where mechanism-specific heuristics explore narrowly and return a single operating point. InferOpt adds a reusable optimization layer with proxy evaluation and budget-aware search, and outperforms three multi-objective search baselines across KV and MoE spaces.

1 Introduction

  • Existing heuristics explore high-dimensional, layer-wise configuration spaces narrowly and yield a single operating point rather than a quality-efficient Pareto frontier.The frontier is needed under varying deployment budgets.Source: §1 · p. 1
  • INFEROPT is an optimization layer that supports continuous KV-memory fractions and discrete routing decisions using variable domains, resource cost, and an evaluation interface.The same search and validation pipeline is reused without mechanism-specific redesign.Source: §1 · p. 1
  • A frozen proxy set, deterministic pre-evaluation resource pruning, and full-benchmark validation form INFEROPT’s three empirically validated components.The admissible budget progressively tightens as feasible observations accumulate.Source: §1 · pp. 1–2
  • Across both KV and MoE spaces, INFEROPT outperforms Random Search, NSGA-II, and MOTPE.Its shared search and validation pipeline is designed to work without mechanism-specific redesign.Source: §1 · p. 2

2 Method

INFEROPT formulates inference configuration as constrained multi-objective optimization over resource cost and quality loss. Its shared pipeline searches frozen proxies, rejects over-budget candidates before evaluation, and re-validates Pareto representatives on the full dataset.

2 Method

  • Resource cost is a closed-form architecture-level proxy—mean retained historical KV blocks or mean activated routed experts—not measured latency, memory, or energy.The optimization also considers quality loss relative to the uncompressed reference on the same examples.Source: §2 · p. 2
  • The pipeline freezes a confidence-aware proxy subset, rejects candidates whose cost exceeds the current budget before evaluation, then re-validates selected proxy-Pareto configurations on the full dataset.Every candidate in search is evaluated on the same proxy examples.Source: §2 · p. 2, Figure 1 · p. 2
  • The KV instantiation searches 28 continuous per-layer retention ratios, while the MoE instantiation searches 26 discrete per-layer expert counts.Apart from variable domains and resource-cost definitions, the search and validation pipeline remains unchanged across the two spaces.Source: §2 · p. 2

3 Experiments

Across KV compression and MoE routing, INFEROPT outperforms the multi-objective search baselines, while the experiments show quality-efficiency gains from searched layer-wise configurations. The evaluation covers proxy and held-out examples, with KV and MoE validation results reported separately.

3 Experiments

  • The experiments use Qwen2.5-7B-Instruct on LongBench for KV compression and DeepSeek-V2-Lite-Chat on MMLU/GSM8K/BBH for MoE routing, with proxy and held-out examples.Search baselines are Random Search, NSGA-II, and MOTPE under the same box constraints, but without adaptive budget-tightening.Source: §3 · p. 3
  • INFEROPT leads all three search baselines: on KV it has the highest mean hypervolume (0.00881) and lowest minimum retention (0.274), while on MoE it is closest to the reference at the lowest budget.The MoE top candidate scores 0.1931, within 0.22 points of the reference; the selected set averages 0.1910, and the budget is P ℓkℓ= 59.3.Source: Table 1 · p. 3, §3 · p. 3

3.1 KV cache compression

  • With the selector fixed, searched KV allocation beats its budget-matched uniform counterpart by 3.21 points with attention scoring and 6.13 points with key-ℓ2.All compressed policies decrease the Full-KV score of 43.83 by 5.79–10.78 points; with allocation fixed, attention scoring exceeds key-ℓ2 by 0.52 points.Source: §3.1 · p. 3
  • At 16K, INFEROPT reduces KV-cache memory from 902.9 to 319.1–321.5 MiB (64.4%) and TPOT from 19.11 ms to 9.83 ms with attention scoring and 10.22 ms with key-ℓ2.The TPOT values are below matched uniform policies (13.34–13.36 ms) and the H2O-style baseline (14.74 ms).Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 MoE top-k routing

  • The MoE schedule assigns k = 6, 3, 1 to the first 9, middle 9, and final 8 layers, reducing routed pairs by 43.0% and TPOT by 19.3% while staying within 0.59 points of the default.At the same budget, it is 0.45 points closer to the default than near-uniform and 0.97 points closer than entropy-based allocation; those controls were post-hoc checks.Source: Table 2 · p. 4, §3.2 · p. 4

4 Conclusion

INFEROPT provides a shared constrained multi-objective search framework for continuous KV-retention and discrete MoE-routing spaces. It outperforms three search baselines in both settings and reports quality-efficiency gains for searched KV and MoE configurations.

  • INFEROPT uses one constrained multi-objective search framework for continuous KV-retention and discrete MoE-routing spaces, without mechanism-specific redesign.The framework requires each mechanism to specify its variable domains, resource-cost function, and evaluation interface.Source: §4 · p. 4
  • Against Random Search, NSGA-II, and MOTPE, INFEROPT outperforms all three in both settings.Source: §4 · p. 4
  • Searched KV allocation improves quality by 3.21–6.13 points over uniform allocation and reduces TPOT by 3.1–3.5 ms, while searched MoE routing cuts routed work by 43.0% within 0.59 points of the default.KV uses 64.4% less cache than Full KV at 16K.Source: §4 · p. 4

A Protocol details

The protocol evaluates KV policies under a shared retention budget and reports the 16K endpoint, while MoE schedules are ranked on a sampled proxy and re-scored on held-out data.

KV compression

  • At 16K context, KV policies share a 128-token protected prefix, 512-token recent window, 128 middle blocks, 32-step compression interval, and mean middle-block retention of 0.321.Compressed policies prune immediately after prefill and at each decoding interval; Full KV is a generation-only baseline.Source: A Protocol details · p. 6
  • KV held-out quality averages 21 LongBench tasks over 1,050 paired examples, with intervals from 1,000 paired bootstrap resamples.The system matrix covers six strategies at 4K, 8K, and 16K, with ten measured repetitions after one warm-up and 128 generated tokens per run.Source: A Protocol details · p. 6

MoE routing

  • The MoE proxy averages a 384-example MMLU/GSM8K/BBH split; its 0.02 quality tolerance labels candidates but is not a hard feasibility constraint.Proxy-to-held-out Kendall correlation is τb = 0.59, and 80% of proxy-front points are preserved.Source: A Protocol details · p. 6
  • MoE reports re-score every schedule on held-out data because the proxy supports only coarse ranking; routed token–expert pairs are an architecture-level work proxy.The three Pℓkℓ = 89 allocations have identical routed-pair counts, so uniform k = 3 is a lower-budget endpoint, not a matched control.Source: A Protocol details · p. 6
Loading 2610.04473v1…