PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal Model Activation Policies for Inference Networks of Large Language Models

Foivos Charalampakos, Md Ibrahim Ibne Alam, Iordanis Koutsopoulos, Koushik Kar

arXiv:2609.15992v1cs.CL

TL;DR

The paper asks how multiple costly LLM experts can be used to balance inference cost and performance without a principled activation strategy. It introduces graph-based inference networks and derives confidence-threshold policies for serial model activation. Experiments and theoretical results show that these policies support target performance while reducing inference cost.

  • Problem

    Existing LLM inference settings lack a principled way to select and conditionally activate available expert models while balancing cost and performance.

  • Method

    The paper models LLMs as graph-based inference networks and optimizes serial activation policies that minimize expected cost subject to a target performance constraint.

  • Results

    The optimal policy has class-specific confidence thresholds for discriminative tasks and a single confidence threshold for generative tasks, with experiments showing reduced inference costs while meeting performance budgets.

  • Takeaways & Limitations

    Confidence-based threshold routing provides an implementable way to use cheaper LLMs first and escalate selectively to more expensive models.

  • Takeaways & Limitations

    The experiments use only a subset of 8,000 samples from each dataset because of computational restrictions.

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have rendered them necessary for Natural Language Processing (NLP) tasks, and their high inference cost motivates the study of cost-performance trade-offs. In practice, several expert LLMs are used in synergy for inference, either in an ensemble mode or in series, yet without a principled approach on how to best use the available models. An adaptive approach can route simple queries to cheaper LLMs and complex ones to more capable, costly models. However, a clear understanding on how to best leverage available expert models is missing. We introduce inference networks, a graph-based framework, where nodes denote different LLMs, and links denote conditional model activations. The inference network design problem is to determine the best topology, namely the best way to use the models that best addresses the cost-performance trade-off. We start from the basic topology of a series of LLM experts, each of which has a different cost and a different level of expertise, which is captured via model confidence. We formulate the problem of optimal activation of these models so as to minimize the expected inference cost subject to a target performance constraint. For this special class of inference networks, we prove that the optimal activation policy has a threshold structure: query the lowest-cost LLM first, and invoke the more expensive LLM only if the confidence falls below a defined threshold. For discriminative tasks, the optimal policy consists of a set of thresholds, one threshold for each class, while for generative tasks, it consists of a single threshold. We provide a structured method to compute the thresholds, and practical confidence estimation mechanisms for both task types. Experiments with open-source LLMs show substantial cost reductions while meeting the specified performance budget.

1 INTRODUCTION

The paper addresses the cost of using powerful LLMs by introducing inference networks that route queries among models according to cost and expertise. For serial networks, it derives confidence-based activation rules that preserve performance while controlling inference cost.

  • Framework: Inference networks model LLMs as directed graphs whose nodes are available models and whose edges represent conditional activations.The design problem is to choose a topology that balances inference cost and performance, with this work focusing on models arranged in series.
  • Framework: The proposed serial policy runs the lower-cost LLM first and uses its output confidence to decide whether to invoke a higher-cost model.This bases routing on model-specific expertise rather than query characteristics alone.
  • Optimal activation: Under mild assumptions, the optimal activation policy is threshold-based: escalate only when the first model’s confidence falls below a defined threshold.For classification, thresholds are class-specific; for generative tasks, the policy uses a single threshold.
  • Optimal activation: The method seeks to minimize expected inference cost subject to a target performance constraint without requiring an externally trained routing model.The paper contrasts this theoretically characterized approach with empirical routing strategies that use auxiliary models.

2 BACKGROUND

The background frames LLM inference as generating answers to natural-language queries across discriminative and generative tasks. It also motivates confidence estimation as the signal needed to judge answer reliability and support adaptive routing.

  • LLM inference: LLMs map natural-language queries to generated answers and support both discriminative and generative NLP tasks.Examples include question answering and text classification, which can be represented as query-answer pairs.
  • Motivation: Modern LLMs achieve strong capabilities across complex NLP tasks, but their tens of billions of parameters impose substantial training and usage costs.This cost motivates choosing among LLM services according to budget and quality expectations.
  • Task types: For discriminative tasks, the answer space can be restricted to predefined classes, whereas generative tasks sample responses from a posterior over the vocabulary.These task formulations determine how model outputs and confidence can be evaluated.
  • Confidence estimation: Confidence estimation assigns a score intended to reflect the reliability of an LLM’s answer, but calculating that score is challenging in practice.Possible approaches include self-assessed or entropy-based uncertainty metrics and external scoring models.

3 SYSTEM MODEL

The system models inference as sequential activation of increasingly costly LLMs, using confidence-based deferral to minimize expected cost under an error constraint. The resulting policies choose the smallest feasible threshold, with class-specific extensions for discriminative tasks.

  • 3 SYSTEM MODEL: The serial network runs the lower-cost LLM first and defers queries to a more complex model when confidence is insufficient.Confidence serves as an indicator of answer reliability and reflects both query difficulty and model expertise.
  • 3.1 GENERATIVE TASKS - SINGLE-THRESHOLD POLICY: The activation problem minimizes expected inference cost subject to a target performance constraint, using the first model’s confidence to control deferral.The threshold policy accepts the small model when β ≥ θ and defers when β < θ.
  • 3.1 GENERATIVE TASKS - SINGLE-THRESHOLD POLICY: For single-threshold policies, the optimum is the smallest feasible threshold because increasing the threshold raises cost while satisfying the error budget.This result holds in the oracle case and remains valid in the non-oracle case even when the feasible set is disconnected.
  • 3.2 DISCRIMINATIVE TASKS - MULTI-THRESHOLDS POLICY: For discriminative tasks, class-specific thresholds form a multi-threshold policy that adapts deferrals to differences in class difficulty and calibration.In the oracle setting, interior classes equalize boundary errors, while easy or difficult classes may always accept or always defer.
  • 3.2 DISCRIMINATIVE TASKS - MULTI-THRESHOLDS POLICY: In the non-oracle discriminative setting, optimal interior thresholds equalize excess boundary error rather than small-model error alone.The relevant quantity compares the small and large models’ class-specific errors, while cost remains coordinate-wise non-decreasing.

4 EXPERIMENTAL RESULTS

Experiments evaluate the policies on classification, question-answering, and machine-translation benchmarks using task-appropriate confidence estimates and cost metrics. The proposed policy maintains target performance while substantially reducing inference cost across generative and discriminative settings.

  • 4 EXPERIMENTAL RESULTS: The study evaluates text classification, question-answering, and machine translation with Monte-Carlo estimates of expected cost and error.Classification uses Accuracy and F1; generation uses embedding cosine similarity and ROUGE-L, with TFLOPs measuring inference cost.
  • 4 EXPERIMENTAL RESULTS: The evaluation compares the proposed policy with full deferral, no deferral, random deferral, FrugalGPT, and HybridLLM.
  • 4.1 GENERATIVE TASKS: Generative experiments use a single threshold based on average token log probability, while classification uses class-specific thresholds based on maximum softmax probability.
  • 4.1 GENERATIVE TASKS: On SQUAD, the policy maintains mLLM performance with a 26.66% reduction in inference cost, while WMT achieves maximal cost savings at the best possible performance.The policy adapts deferral to task difficulty rather than using static or random allocation.
  • 4.2 DISCRIMINATIVE TASKS: ∼82% cost savings on SST-2 accompany matched full-deferral accuracy, while AGNEWS and EMOTION achieve full-deferral performance with ∼16% and ∼68% savings.
  • 4.2 DISCRIMINATIVE TASKS: On FAKENEWS, performance degrades only ∼2% from full-deferral while inference cost falls by ∼45%.

5 CONCLUSION

The paper frames LLM inference as graph-structured conditional activation and formalizes optimal activation under a cost-performance constraint. Its experiments show that the resulting policy can reduce cost while preserving near-optimal classification performance.

  • 5 CONCLUSION: Inference networks represent alternative LLMs as graph nodes and conditional model activations as edges.
  • 5 CONCLUSION: Optimal activation minimizes expected inference cost subject to a target performance constraint, with threshold policies characterized for the two-model setting.The policy uses class-specific thresholds for discriminative tasks and a single threshold for generative tasks.
  • 5 CONCLUSION: Classification experiments with GEMMA3-1B and GEMMA3-4B show near-optimal performance with significant inference-cost reductions across datasets.

Models (Supplementary Material)

The supplementary material covers related model-chaining and routing approaches, proofs of threshold optimality, computation of optimal thresholds, and edge cases affecting uniqueness.

  • Models (Supplementary Material): Prior model-chaining methods sequentially escalate queries, while routing methods assign each query to one model in advance.
  • Models (Supplementary Material): The optimal threshold is the leftmost feasible point when expected cost is non-decreasing and the error constraint defines a feasible right ray.Under strict monotonicity, the optimizer is unique and the constraint is tight.
  • Models (Supplementary Material): Ties arise only when the error function is flat at the budget level over an interval beginning at the smallest feasible threshold.
  • Models (Supplementary Material): If the initial threshold already satisfies the budget, θ∗= 0; when the confidence density vanishes near zero, every threshold in that interval can be optimal.
  • Models (Supplementary Material): Analytic threshold computation is available under closed-form confidence-density and error-rate assumptions, while general cases can use numerical solution methods.The supplementary material gives a uniform-confidence, linearly decreasing-error example and notes extensions to arbitrary intervals and distribution families.
  • Models (Supplementary Material): Calibration can generally ensure the confidence assumptions asymptotically under proper scoring rules when there is no distributional shift.

B.3 MULTI-THRESHOLDS ORACLE CASE

In the oracle multi-threshold setting, optimality equalizes boundary errors across interior classes while satisfying the performance budget, with possible non-uniqueness on flat regions.

  • Optimality condition: At any interior optimum, all active classes share a common boundary error level, while boundary classes may saturate at 0 or 1.The tightness condition determines the common level unless coordinates saturate.
  • Non-uniqueness: The optimal threshold vector need not be unique when error or budget mappings contain flat segments, including zero-density plateaus and cross-class indifference.These conditions permit threshold perturbations without changing cost or error.
  • Computation: Optimal thresholds can be computed by solving a scalar budget equation, using numerical root-finding when closed-form inversion is unavailable.The appendix identifies bisection or Brent’s method for the resulting equation.
  • Analytic example: Under uniform confidence densities and linear class-specific error functions, Lagrange multipliers yield an analytic threshold system whose general solution may still require numerical solution.Symmetric special cases reduce to a quadratic expression and recover the single-threshold solution when δ = 1.
  • Illustrative comparison: The multi-threshold policy can improve accuracy over a single threshold in the illustrative binary classification setting.The figure compares accuracy and cost as functions of threshold choice.

C.3 BAYES-OPTIMAL POLICY

The Bayes-optimal policy minimizes a Lagrangian by choosing, pointwise, between accepting the small model and deferring to the master model according to their conditional risks.

  • Bayes-optimal decision: The Bayes rule accepts the small model when its conditional risk is no greater than the master model’s risk, and otherwise defers.This pointwise rule is globally optimal for the Lagrangian objective.
  • Risk comparison: Acceptance is optimal when the small model’s excess error is at most the cost-normalized margin cm/λ.The excess error is the small model’s error minus the master model’s expected error.
  • Constrained optimization: The constrained error–cost problem can be characterized through Lagrangian minimizers because Pareto-optimal risk–cost points admit a suitable multiplier λ.Thus, analyzing the pointwise Bayes rule identifies policies on the relevant frontier.

C.6 FROM ERROR THRESHOLD TO CONFIDENCE-THRESHOLD

Under monotone risk, the Bayes acceptance region becomes a confidence-threshold policy, with thresholds obtained by inverting error curves and calibrating a scalar multiplier to the target budget.

  • Threshold structure: Monotone risk makes each class’s acceptance set an interval beginning at a confidence threshold, yielding per-class single-threshold policies.The threshold may include or exclude its boundary point without changing Bayes risk.
  • Multi-class optimality: At an interior optimum, the excess error at the thresholds is equalized across active classes; in the oracle case this reduces to equal boundary errors.The non-oracle condition includes the master model’s class-specific error.
  • Threshold calculation: The threshold for class k is found by inverting its error curve at the common excess-error level ε̄m,k + cm/λ, with endpoint saturation when no interior solution exists.This accommodates always-accept and always-defer cases.
  • Budget calibration: A scalar root-finding step selects λ so the resulting policy meets the target error budget.Bisection or Brent’s method can solve the monotone budget equation.
  • General policy class: The framework allows arbitrary measurable policies initially, then establishes threshold optimality under the monotone-risk assumption.The same approach applies to class-conditional and binary confidence settings.

D.2 BAYES-OPTIMAL DEFERRAL RULE IN THE BINARY CASE

For binary deferral, the Bayes-optimal action compares conditional risks pointwise; monotone confidence risk converts this rule into a single threshold in the confidence score.

  • Bayes rule: For fixed λ, the Bayes-optimal rule accepts or defers according to which action has lower conditional Lagrangian risk.The rule is globally optimal because any alternative policy has no lower pointwise risk.
  • Deferral condition: The rule accepts the small model when its excess error over the master model is at most cm/λ, and defers otherwise.This expresses deferral as paying additional cost when the small model is sufficiently worse.
  • Threshold structure: Under monotone risk, higher confidence corresponds to no greater small-model error, so the acceptance region is an interval above a single confidence threshold.The threshold representation follows from the interval structure of sublevel sets.

E CONTINUITY OF THE CONSTRAINT FUNCTION

The section proves continuity of the single- and multi-threshold error functions, supporting boundary-based optimization of the activation constraints.

  • Optimization implication: The continuity results establish the regularity needed for the section’s constrained threshold optimization arguments.The proof proceeds by showing that the relevant integral components are absolutely continuous before summing them.
  • Single-threshold continuity: The single-threshold oracle and non-oracle error maps are absolutely continuous, and therefore continuous, on [0, 1].The proof uses integrability of the density-weighted error function and absolute continuity of tail integrals.
  • Multi-threshold continuity: For multi-threshold policies, the oracle and non-oracle error maps are continuous on [0, 1]^K and absolutely continuous in each coordinate.Class-wise error contributions are combined using class probabilities and coordinate-wise absolute continuity.

F.1 DATASETS

The appendix describes datasets spanning binary and multiclass classification, question answering, and machine translation, alongside routing and evaluation procedures.

  • Datasets: Five NLP datasets cover binary classification, multiclass classification, and free-form question answering.The datasets are SST-2, FAKENEWS, AGNEWS, EMOTION, and SQUAD.
  • Datasets: SST-2 and FAKENEWS support binary classification, while AGNEWS and EMOTION support multiclass classification.AGNEWS uses four topic labels, and EMOTION uses six emotion categories.
  • Datasets: SQUAD is evaluated as free-form reading comprehension rather than being restricted to fixed answer spans or labels.The dataset pairs questions with Wikipedia context passages, while the evaluation permits open-ended answers.
  • Evaluation procedure: The appendix includes comparisons with alternative policies and model pairs, including the GEMMA3-4B/GEMMA3-12B classification pair.Oracle and non-oracle error formulations are both specified for the threshold policies.
  • Evaluation procedure: The empirical procedure estimates expected error and cost with Monte Carlo sample averages under single- or class-conditional threshold policies.Single thresholds are used for open-ended generation, while multi-threshold vectors apply thresholds by predicted class.

F.4.3 Finite-sample effects and empirical feasibility

Finite samples make threshold optimization discrete: empirical error and deferral functions are stepwise, so exact feasibility and algorithmic complexity depend on the observed confidence values.

  • Finite-sample effects: Finite-sample error and deferral rates are stepwise because accept/defer sets change only when thresholds cross observed confidence values.This discretization can prevent an empirical solution from attaining the error budget exactly.
  • Single-threshold optimization: The single-threshold optimizer selects the smallest observed confidence cutoff satisfying the empirical error constraint, with tie-aware sorting scans requiring O(N log N) time.Candidate thresholds are the distinct confidence values plus accept-all and defer-all endpoints.
  • Single-threshold optimization: In the oracle single-threshold case, BFPRT can find the relevant confidence quantile in O(N) worst-case time without full sorting.This guarantee generally does not extend to the non-oracle case because deferred-model errors also enter the constraint.
  • Multi-threshold optimization: Multi-threshold optimization forms one deferral/error option per class and chooses one option per class under a global error-count budget.Per-class option construction sorts confidence values within classes, with total options U ≤ N + K.
  • Multi-threshold optimization: Dynamic programming gives an exact finite-sample solution but can require pseudo-polynomial time, while the Lagrangian sweep is polynomial but not guaranteed globally optimal.The dynamic program treats the problem as a multi-choice knapsack instance; the Lagrangian method uses the option-matrix size.

G SUFFICIENCY OF THE CONFIDENCE SCORE

The section formalizes when confidence is sufficient for routing and reports preliminary tests in which adding richer query information did not significantly improve prediction of which model would be better.

  • Confidence-based routing: The confidence score is defined as a scalar signal for deciding whether to accept the small model or defer to the larger model.The action losses compare accepting the small model against paying additional cost and using the larger model.
  • Theoretical sufficiency: If confidence is a sufficient scalar summary and the model-advantage function is non-increasing in confidence, the globally optimal policy is a confidence threshold.No policy using the full gating information can outperform the best policy using sufficient confidence alone.
  • Empirical sufficiency: Preliminary sufficiency experiments compared confidence-only predictors with predictors adding text embeddings or logits statistics, using cross-fitting and permutation tests.The comparison evaluates whether richer information significantly reduces prediction error beyond confidence.
  • Empirical sufficiency: The preliminary experiments found no statistically significant error difference, so the study used confidence as the deferral signal.This result was reported for the tested model pairs and additional information types.
Loading 2609.15992v1…