Which LLM for Which Work? Budgeted Model Allocation under Uncertain Evaluation

Hamed Khosravi; Xiaoming Huo

Source-linked AI summary

Which LLM for Which Work? Budgeted Model Allocation under Uncertain Evaluation

Hamed Khosravi, Xiaoming Huo

arXiv:2608.29560v1Aug 30, 2026cs.LGmath.OCstat.ML

TL;DR

The paper studies budgeted assignment of LLMs to workloads when quality tables are uncertain because assignment is selective and evaluation scores are imperfect proxies. It constructs evidence-consistent uncertainty sets, certifies decisions with an estimated-table and adverse-table solve, and uses CASE to target further readouts when they disagree. Experiments show that correcting assignment bias can leave substantial measurement-related loss, while available evidence often does not determine the allocation.

Problem
LLM allocation is routine when the quality table is known, but selective assignment and proxy evaluation make the table’s entries uncertain in different ways.
Method
The paper bounds evidence-consistent quality tables, tests the current budgeted allocation against a least-favourable table, and uses CASE to target randomized readouts where allocations disagree.
Results
With selective routing, CASE reduces target-value loss at 2×, 4×, and 8× the cheapest feasible spend, while randomized assignment removes its improvement over trusting the log.
Takeaways & Limitations
The evidence often does not determine the allocation, so targeted information about model quality can matter more than further optimizing the same uncertain estimates.
Takeaways & Limitations
CASE certifies robustness only relative to its supplied uncertainty set, while the true table may lie outside it and row-level correction can miss cell-specific residual bias.

Abstract

from arXiv · show

A company with a fixed artificial intelligence (AI) budget must decide which large language model (LLM) handles each recurring workload. What it lacks is the quality table, how well each model performs on each workload. Given that table, the decision is a multiple-choice knapsack problem and is routine to solve, so estimating it is the difficulty, and that estimation fails in two ways. Models are rarely compared on the same work, and the recorded score is usually a proxy rather than the outcome the company values. Causal and off-policy methods repair the first but condition on the second, while evaluator-validation methods estimate the second but stop short of the decision. Worse, buying more re-evaluation cannot settle the second: randomization governs which requests are scored, not how a score is produced, so the table stays uncertain however much evaluation is purchased. Yet the deployment decision may still be determined even when the table is not. We therefore ask whether one assignment stays optimal across every quality table consistent with the evidence. For the fixed-budget problem, this admits an exact two-solve certificate: solve once at the estimated table and once at a least-favourable table. Agreement certifies the assignment; disagreement identifies the model-workload pairs where further evidence can matter. We propose CASE (causal active sequential experimentation), which targets evaluation to those pairs and repeats the test as evidence accumulates. On a production log, the measurement failure is the larger of the two: correcting assignment exactly still leaves most of the loss, and randomized re-evaluation does not remove it. In our experiments, the available evidence often does not determine the assignment. On paid software tasks, better information about model quality yields more savings than further optimization of the assignment on the same estimates.

1 Introduction

LLM allocation is easy once the quality table is known, but estimating that table is difficult because assignment is selective and recorded scores are imperfect proxies. The paper therefore asks whether evidence can determine the deployment decision without determining the table itself.

Motivation: A company must assign one model to each workload under a shared budget because prices and model strengths vary across work types.The motivating workloads include coding, support, document summarization, and sales outreach.
Problem: Given a known quality table, the allocation is a standard multiple-choice knapsack problem; estimating the table is the central difficulty.The table records each model’s average designated outcome for each workload.
Measurement challenges: Production assignment can confound quality comparisons, while evaluators can systematically depart from the designated outcome.A cheaper model may appear competitive if it receives easier requests, and scores can depend on answer length, position, or model identity.
Measurement challenges: Randomized re-evaluation removes selective-assignment bias but does not remove evaluator bias, because it randomizes which requests are scored rather than how scores are produced.These are distinct validity failures and respond differently to additional evaluation spending.
Approach: The paper tests whether one allocation remains optimal across every quality table consistent with the evidence, rather than requiring a precise table estimate.For the budgeted rule, agreement between an estimated-table solve and a least-favourable-table solve certifies the allocation; disagreement localizes useful further evidence.
Related work: Existing approaches address only parts of the problem: uncertainty optimization assumes the uncertainty set, while causal and off-policy methods take recorded scores at face value.Evaluator-validation methods estimate scoring quality but do not directly settle the allocation decision.

2 Problem formulation

The paper formalizes workload-to-model assignment as choosing one supported model per workload under either a serving budget or per-workload admissibility rule. Its decision map is piecewise constant, so only within-workload quality differences and boundaries between allocation regions affect the decision.

Problem setup: The operator assigns one model to each workload using known prices, traffic shares, serving volume, and budget; the quality-coefficient table is the only unknown input.Each workload has a traffic share, each model has a per-request cost, and only supported models may be selected.
Budgeted allocation: Under the budgeted rule, binary selections maximize total quality subject to choosing one model per workload and respecting a shared spending cap.The shared budget couples workload choices, so they cannot be optimized independently.
Admissibility allocation: Under the admissibility rule, the operator chooses the cheapest model within δ of each workload’s best quality, without a budget constraint.This rule decomposes by workload and is not a special case of the budgeted rule.
Decision map: A fixed tie-breaking order makes the decision map Ψ single-valued even when multiple allocations are optimal.The paper uses cheapest-first and then lexicographic tie-breaking.
Decision regions: The decision map is piecewise constant: small table errors below its stability radius leave the allocation unchanged, while region boundaries mark possible assignment changes.Budget coupling creates global hyperplane boundaries, whereas the decomposable rule has workload-specific quality-gap thresholds.
Decision-relevant error: Column-level quality shifts do not affect allocations, but row-level shifts can change model rankings and therefore the decision.Only within-workload error spreads are decision relevant because common column shifts alter all feasible allocations equally.
Uncertainty and loss: The paper defines target-value loss relative to the allocation chosen under the true quality table, while practical decisions must instead use an evidence-supported set of possible tables.Settlement asks whether that set lies inside a single allocation region.

3 Proposed approach

The proposed method builds an uncertainty set from production logs, randomized readouts, and evaluator bands, then tests whether the current allocation remains optimal throughout that set. If not, CASE buys evidence only in cells separating the current allocation from its closest competitor and repeats the test.

Evidence construction: Production logs provide selective averages M_kj, while randomized re-evaluation provides causal averages R_kj for purchased model–workload readouts.A readout is one randomized observation of a cell (k, j).
Evidence construction: Randomized re-evaluation corrects assignment bias by comparing logged and randomized averages, and the observed differences are aggregated into model-level corrections.The correction assumes evaluator error is the same on average in both settings.
Uncertainty set: The remaining measurement error is assumed to lie within a workload-specific evaluator band η_j, because the correction does not estimate that residual bias.This assumption supplies the uncertainty width rather than resolving evaluator validity.
Uncertainty set: The uncertainty set treats each cell’s true value as lying between its randomized score minus η_j and its randomized score.The interval can be represented by a centre and half-width because common column shifts do not change the decision.
Settlement test: Settlement means that the current optimal allocation remains optimal for every plausible table, allowing ties; for the budgeted rule, this is equivalent to remaining optimal at one adverse table.The adverse table lowers selected cells and raises unselected cells to make the current allocation’s margin as small as possible against every rival.
Settlement test: The exact certificate requires two solves of the same budgeted allocation problem: one at the current estimate and one at the least-favourable table.The proposition states that agreement certifies settlement over the uncertainty set.
Active experimentation: When the test fails, CASE identifies the closest competing allocation and purchases an unread readout from either the differing alternative or current model cell.A readout shifts the uncertainty-set centre and margins but does not narrow the cell’s evaluator-band width.
Active experimentation: CASE repeats estimation, adverse testing, and targeted purchasing until remaining uncertainty cannot change the allocation, without reading every cell.The procedure therefore targets evidence where it can still affect the decision.

4 Experiments

Experiments show that correcting selective routing improves decisions but leaves substantial evaluator-induced uncertainty. The exact certificate rarely settles allocations at real evaluator widths, while better quality information can be more valuable than optimizing assignments from inaccurate estimates.

When the range determines the allocation: Only 1 of 210 LLMRouterBench partition–budget instances is settled at the measured evaluator width, making evidence rather than optimization the binding constraint.At η = 0.02, 21 of 30 partitions are determined at 1.25× the cheapest feasible spend, but none are determined at 8×.
Production-log evaluation: Cellwise-perfect assignment repair still leaves 2.01 × 10^-3 loss, because removing selection bias exposes mismatch between click scores and like outcomes.The router’s model preferences and click-score preferences partly cancel before selection correction.
Evaluator uncertainty: The measured evaluator spread is substantial: KuaiRand has median η = 0.128 across workloads, while LMArena judges have spread 0.336.The KuaiRand range is 0.095 to 0.173 and exceeds the η = 0.02 used in benchmark replays.
Randomized re-evaluation: Under faithful scoring, CASE reduces loss from 0.0499 to 0.0338, while unlimited audit information reaches 0.0318, capturing 89% of that correction with a 35% evidence budget.With randomized assignment, CASE no longer improves on trusting the log; distorted evaluation leaves additional loss under either assignment scheme.
Value of better information: On 150 paid SWE-Lancer tasks, observable-table routing gains about $1,000 over the best single model, while task-level realized-outcome oracle routing gains a further $35,000.The oracle comparison bounds the value of better quality information rather than measuring it directly, because it decides per task instead of per payout band.

5 Conclusion

The paper frames trustworthy quality information as the binding constraint in budgeted LLM allocation and introduces a certificate that may decline to settle decisions under measured evaluator uncertainty.

Conclusion: Two solves are enough to test whether all evidence-consistent quality tables imply the same budgeted allocation.Agreement settles the decision; disagreement identifies model–workload pairs where additional evidence can matter.
Conclusion: At measured evaluator bands, the method often declines to certify, indicating that trustworthy quality information—not optimization—is the binding constraint.
Conclusion: The implementation is accompanied by a reproducibility repository for the logged LLM comparison audit.

A Notation

The notation distinguishes true target quality from logged and randomized measurements, while Figure 2 illustrates how quality entries determine budgeted allocations.

A Notation: Table 3 orders the symbols introduced in Sections 2 and 3.
A Notation: Only M and R are observable, whereas µref represents the reference quality table and C denotes evaluator error used in the appendices.
A Notation: Figure 2 uses model rows, workload columns, prices, volumes, quality cells, and outlines to depict an illustrative allocation.

B Proofs

The appendices formalize the two decision rules, prove the least-favourable-table certificate, and characterize evaluator assumptions, ambiguity, and limitations.

B Proofs: Assignment recovery assumes equal evaluator mean error across observational and randomized streams, while model-level evaluator error is assumed constant across workloads.
B Proofs: Under model-level evaluator error, the logged-minus-randomized difference cancels evaluator distortion but cannot remove model-level evaluator preference.
B Proofs: The two decision rules are not equivalent: quality-constrained selection can choose a model unreachable by scalarization and can be suboptimal at its own spend.
B Proofs: The budgeted rule’s feasible set depends on prices and traffic, while its objective compares quality across workload assignments.
B Proofs: The proof relies on a product of intervals, allowing each differing cell to attain its own worst-case extreme independently.
B Proofs: A least-favourable table lowers the incumbent cells and raises alternative cells, so one second solve exactly tests settlement over the entire interval uncertainty set.
B Proofs: The quality-constrained rule is point-identified when a universally admissible model exists and no cheaper ambiguous model precedes it.

D Experimental details and additional results

Additional experiments distinguish selective-routing correction from evaluator distortion and warn that results from the two decision rules are not directly comparable.

D Experimental details and additional results: Randomized evidence repairs selective-routing bias, while evaluator error remains as uncertainty around µref that can cross a decision boundary.
D Experimental details and additional results: The margin sweep uses the budgeted allocation, whereas other runs and the production-log replay use a per-workload argmax without a shared budget.
D Experimental details and additional results: Levels are not comparable cell by cell across the two rules, so the additional results report only directional robustness.

D.1 Separating what randomized evidence can and cannot repair

Randomized evidence can repair selective assignment, but it cannot remove the loss caused by a proxy evaluator. The remaining measurement mismatch persists even with stronger correction or more audits.

Correction limits: The cellwise repair reaches zero target-value loss under faithful scoring but leaves 0.0030±0.0006 under distorted scoring.More randomized evidence can approach the correction-family ceiling, but cannot remove evaluator-measurement error.
Correction limits: On RouterBench, faithful-scoring loss falls from 0.0784 to 0.0421, while distorted scoring leaves 0.0053 after cellwise-perfect assignment repair.Doubling distortion raises the wrong-model floor from 18.3% to 30.6% and remaining target-value loss from 0.0030 to 0.0069±0.0014.
Scope: The acquisition-rule ordering reverses with a larger candidate set, so the paper excludes that secondary effect from its main claim.The cost of auditing under matched-random assignment also changes sign.
Alternative repairs: Exact-propensity methods can achieve decision quality comparable to CASE without audit spend, but the predeclared CASE comparison remains unresolved.The comparison is −0.0023 with CI [−0.0100, +0.0055].

D.2 When the remaining range determines the allocation

The allocation is determined when the remaining quality range fits inside one decision region, not when the quality table is estimated exactly. Empirical margins shrink as uncertainty widens, with behavior depending on the serving budget.

Settlement criterion: Settlement requires the remaining uncertainty width to fit inside one decision region after selective-routing correction.The stopping rule tests whether an adverse solve changes the incumbent allocation.
Budget sweep: On LLMRouterBench, the median tolerated spread falls from 0.0456 at 1.25× the cheapest feasible spend to 0.0026 at 8×.At η = 0.02, 21 of 30 partitions remain determined at the tight budget and none at the loose one.
Budget sweep: At 12×, the LLMRouterBench median margin rises because the serving budget stops binding on 12 of 30 partitions.RouterBench binds at every budget and shows no trend.
Uncertainty sets: The model-level uncertainty set is a strict subset of the free box, and certifying over it requires the 2K vertices of {Ck}.The two sets coincide when the binding rival differs from the incumbent on one workload, separating only for multi-workload conflicts.
Robustness: Confirmatory benefits of randomized evidence under selective routing remain separated from zero across partition, request, and benchmark-slice resampling.The value-scale measurement floor is weaker under the five-slice cluster bootstrap.

D.3 Measuring the remaining width on real data

Real-data experiments show that randomized readouts add limited improvement when selection error is small, while evaluator disagreement leaves substantial uncertainty. Repeating a scoring channel improves precision without validating the target outcome.

Experimental design: The controlled replay separates routing and scoring by crossing selective or matched-random assignment with faithful or distorted scoring.Fit, audit, and scoring cohorts are disjoint, with paired assignment conditions using the same per-cell counts.
Production replay: Increasing nread from 30 to 100 to 300 changes traffic-weighted target-value loss only from 0.00157 to 0.00148 to 0.00149.Randomized evidence changes the routing component of information about µref, while evaluator uncertainty remains as width in U.
Evaluator validity: Repeating the same scoring channel can reduce harmful switches and increase precision, but does not establish that it measures µref.The evaluator still disagrees with the designated outcome on 4.6% of contested workloads and 8.3% under stronger distortion.
Evaluator validity: On LMArena, gpt-4o-mini agrees with human votes on 0.430, flips after answer-order swaps on 0.335, and prefers longer answers at +0.243 versus +0.130 for human verdicts.A two-way decomposition attributes 53.4% of its error to model main effects, compared with a permutation-null median of 13.0%.

D.4 What better information about the quality table is worth

Better information about the designated quality outcome can change deployment value more than optimizing assignment on the same estimates. The economic comparison makes the remaining measurement gap concrete.

Value of information: On LLMRouterBench, target quality rises from 0.4798 for the raw log to 0.4924 after selection adjustment and 0.5255 with the designated outcome.The randomized audit costs about $2.39 at the 35% evidence budget on LLMRouterBench and $0.89 on RouterBench.
Economic scale: On 150 SWE-Lancer managerial tasks, observable-estimate routing costs $70,625, while realized-outcome routing costs $35,625.The roughly $35,000 gap is the calculated economic value of information missing from the observable quality estimate.
Decision framing: The quality-information set is derived from production logs, randomized re-evaluation, and remaining measurement ambiguity rather than known in advance.The paper asks whether the current allocation is invariant throughout that evidence-derived set.