PaperlayerView exact revision on arXiv

Source-linked AI summary

Most of the LLM Routing Gap Is Task Type

Janghoon Lee

arXiv:2608.23023v2cs.CL

TL;DR

The paper investigates what accounts for the gap between LLM routing and the oracle. It evaluates a complete two-run matrix across models, task types, and languages, finding that static task and language structure recovers most of the reproducible opportunity while rerun movement is larger than the remainder. The resulting static table scores higher and costs less than the best single model, but the comparison is in-sample.

  • Problem

    Existing routers remain far below the oracle, leaving unresolved what the routing gap consists of in coordinates a policy can condition on.

  • Method

    The paper runs 14 models on all 294 queries across seven task types and three languages twice, then decomposes the oracle-minus-best-single gap by task type and language.

  • Results

    Task type recovers 21 of 29 both-run gap items and language recovers 2 more; the static table scores 262 of 294 items at $3.332910 per run versus Claude Opus 5’s 245 at $7.688898.

  • Takeaways & Limitations

    On this matrix, 23 of 29 improvable items are visible from task type and language, so a fixed table captures most of the measured opportunity before learned routing.

  • Takeaways & Limitations

    The table is fitted and scored on the same 294 items, making 262/294 an in-sample upper-bound estimate rather than an out-of-sample performance claim.

Abstract

from arXiv · show

An LLM router picks which model should answer each query. The appeal is that models fail on different questions. Whatever single model is best overall still gets some wrong, and another model in the pool gets many of those right. Getting that choice right every time is the ceiling, and a router is an attempt to approach it. However, recent work reports that routers do not get close. Across 21 routing methods on five benchmarks, sharply different designs land within a fraction of a point of each other, and all of them stay far below that ceiling. Learned routers often fail to beat simply always calling the strongest model. We ask what those missed questions have in common. We set fourteen models to answer all 294 questions, with 7 task types across 3 languages: Korean, English and Hindi. We ran the whole matrix twice, changing nothing, but 5.37% of the 4,116 model-question pairs came out scored differently anyway. Run-to-run movement like that is normal, and we argue that a small win does not show that routing did anything, ours or anyone else's. Counting an answer correct only when the model got it right in both runs, 29 questions on this matrix can be improved with routing. Every correct-answer count here is on that rule. Task type accounts for most of them: assigning each task type one model in advance, chosen once and never updated, improves 21 of the 29. Splitting each task type by language improves 2 more and leaves 6 of 294 unoptimized. That handful is what a learned router would have been built for, and it is smaller than the run-to-run movement above, which is a share of pairs rather than of questions. The static table we adopted answers 262 of 294 questions at \$3.33 per run, against the best single model's 245 at \$7.69. All of this is fitted and scored on the same 294 questions with no holdout.

1 Introduction

The paper asks what explains the gap between routing performance and the oracle, focusing on observable task type and language structure. On this matrix, static choices recover most of the reproducible gap, while rerun movement exceeds the remaining opportunity.

  • Motivation: Across 21 routing methods on five benchmarks, the top methods converge within a narrow performance band while remaining 10 to 30 percentage points below the oracle.The top five differ by 0.22 percentage points on average, and the top fifteen on RouterBench by 0.23 percentage points.
  • Task type and language: 21 of 29 both-run oracle-minus-best-single items are recovered by assigning one model to each task type.The corresponding single-run split is 19 of 31 items, or 61.3%.
  • Task type and language: 2 additional items are recovered by splitting task types by language, leaving 6 of 294 items under the both-run rule.The single-run residual is 9 of 294 items, or 3.06%.
  • Run-to-run movement: 5.37% of model-query cells change scored correctness across identical reruns, exceeding the remaining routing opportunity on this matrix.The rerun movement is 221 of 4,116 model-query cells; the residual is measured over items, a different denominator.
  • Scope: The study’s scope is limited to 14 models, seven task types, three languages, two runs, one gateway, and an in-sample policy comparison.The authors defend a narrow statement about this matrix rather than claiming routers are unnecessary.

2 Setup

The setup is a complete, replicated matrix in which 14 models answer 294 queries spanning seven task types and three languages. Scores use task-specific binary rules, while costs and model selection are recorded from the configured gateway and catalog.

  • Matrix design: 294 queries combine seven task types, three languages, and 14 items per task×language cell; every model answers every query.The matrix contains 4,116 model-query cells per run and 8,232 keys across two runs.
  • Matrix design: 8,229 of 8,232 run keys are scored after three infrastructure failures on the same Korean coding item.The failures reduce the affected model’s denominator where they occur.
  • Costs: The configured cost axis uses billed usage cost, while listed prices are USD per million tokens from a catalog snapshot and may carry conditions.Conditional prices do not enter reported accuracy or cost figures for the stated reason.
  • Task types and sources: The seven task types correspond to seven single-source partitions, making task type and source the same partition on this matrix.Sampling is seeded and stratified where the source supports it.
  • Languages: The three language conditions are Korean, English, and Hindi, with one shared item identifier across languages.The design uses 98 unique item identifiers multiplied by three languages to produce 294 queries.
  • Languages: Six task types are translated from English by one pipeline, while knowledge uses an existing parallel Korean-English-Hindi release.Numbers, formulas, code, tests, identifiers, JSON keys, and other specified fields remain in English by construction.
  • Scoring: The primary denominator is all 294 items, with parse failures and cap hits scored as incorrect.Diagnostic parse-ok and cap-hit denominators are reported only as sensitivity variants.

3 Execution noise floor

Repeating the same matrix under identical settings changed 5.37% of scored model-query cells, establishing a substantial execution-noise reference. The paper therefore compares accuracy differences against multiple references, while acknowledging that the procedure itself is thin and not preregistered.

  • 3.1 What re-execution moves: 221 of 4,116 model-query cells flipped their scored-correct bit between identical executions, a 5.37% movement rate.The runs used the same queries, temperature 0, and seed 42; 115 flips were 1→0 and 106 were 0→1.
  • 3.1 What re-execution moves: $51.557488 versus $49.591932 produced a −3.81% whole-matrix billed-cost movement between the two executions.This is the cost-axis reference for later billed comparisons and is a different estimand from correct-bit flips.
  • 3.1 What re-execution moves: At least 9 of 294 items, or 3.06%, changed from correct to incorrect when DeepSeek V4 Flash was rerun after an output-cap change.This per-item reference changes configuration as well as re-executing the model, so it is weaker than the identical-configuration A/B pair.
  • 3.2 How a difference is read against these references: Accuracy differences are counted in items over the relevant item set, while billed-cost differences are measured as shares of the costlier policy’s bill.The paper warns that rates on unlike denominators may be compared only as one rate standing above or below another, not converted into a rate difference.
  • 3.2 How a difference is read against these references: A difference at or below any applicable reference is reported as unresolved, whereas a difference above every applicable reference is reported as resolved.Resolved means only that the difference exceeds observed re-execution movement; it does not establish persistence on another item set, model pool, or third run.
  • 3.2 How a difference is read against these references: The comparison procedure is not preregistered and rests on only two executions, so its references carry uncertainty from n = 2.The paper reports sizes and directions so readers can apply a different threshold to the same quantities.

4 Gap decomposition

The oracle-minus-best-single gap is primarily recoverable through static task-type assignments, with a smaller residual inside task types and little additional gain from language. Under the paper’s both-run rule, task type recovers 21 of 29 gap items, task×language leaves 6 unresolved items, and the adopted lookup is a separate 262-item policy.

  • 4.3 Where the 29 items sit: The figure partitions each oracle-to-best-single bar into between-type recovery and residual within-task-type recovery under both scoring rules.The bar percentage is the between-type share of the total, and the terminal count is the total gap.
  • 4.3 Where the 29 items sit: 21 of 29 both-run gap items, or 72.4%, are recovered by assigning each task type its own best model.The remaining 8 items require a different model within a task type; 7 of those 8 are coding.
  • 4.3 Where the 29 items sit: Coding supplies most of the within-task-type residual, while the other six task types together contribute only 3 of 294 items, or 1.02%, in the single-run decomposition.Coding is the dominant within-type residual in Korean, English, and Hindi, though its magnitudes differ by language.
  • 4.3 Where the 29 items sit: Adding language after task type recovers 2 more both-run items and leaves 6 of 294 items, or 2.04%, unresolved.The residual is below both accuracy references, so this procedure does not resolve it from observed re-execution movement.
  • 4.3 Where the 29 items sit: The adopted lookup answers 262 of 294 items, not 268, because post-result coding overrides and cross-run tie handling distinguish it from the in-sample argmax.Its own residual is 12 of 294 items and is reported as a separate §6 figure.
  • 4.3 Where the 29 items sit: The task-type result is static structure plus a coding residual, not evidence that a learned router is useful.The gate only determined whether the oracle gap was large enough to consider routing; the decomposition determined the paper’s interpretation.

5 On this matrix, which model is best depends on the scoring rule

On this matrix, the best single model changes with the scoring rule, and the observed margins are too small to support a model-independent winner. The analysis therefore treats the scoring rule as decisive and reports competing variants rather than claiming systematic unfairness.

  • Model identity depends on the rule: Five different models lead across ten scoring rules, so “best model” has no single answer on this matrix.The rules include scoring variants and per-language strata.
  • Margins are small: No top-two margin exceeds 0.0272 in accuracy, with the largest shared-denominator margin equal to 8/294 = 2.72%.The four remaining margins use per-model exclusion denominators and are reported as accuracy-point sizes rather than item rates.
  • Margins are smaller than re-execution movement: 221 of 4,116 model-query cells flip scored correctness between identical runs, a 5.37% movement larger than the main one-item margin.The same model, Opus, moves from 249/294 to 252/294 across the two runs.
  • Interpretation: The supported conclusion is narrow: the matrix identifies winners under stated scoring rules, not a universally best model or systematic scoring-rule bias.The analysis does not explain why individual models perform better under one rule than another.
  • Two apparent rule effects: The cap_hit exclusion promotes Qwen3.7 Plus, but the cap is an experimental parameter rather than an inherent scoring-rule property.The paper reports both variants instead of deciding whether capped items reflect ability or token-limit encounters.
  • Two apparent rule effects: The both-run rule promotes Gemini 3.7 Flash because it flips zero correct bits, making its ranking partly a determinism ranking.This follows from requiring correctness in both runs, not from evidence that the rule is systematically unfair.

6 A static task×language lookup

The paper adopts a static table that selects one model for each task×language cell, using both-run accuracy and cost-aware tie rules. The resulting lookup improves accuracy and cost relative to the best single model on this in-sample matrix, while a cheaper alternative leaves the accuracy difference unresolved.

  • Policy construction: 21 task×language cells receive one model selected in advance, with both-run accuracy maximized and billed cost breaking accuracy ties.The policy uses a static lookup rather than a learned router.
  • Policy construction: 267/294 @ $3.368173 is the intermediate policy after replacing the coding-Hindi choice with Grok 4.3.The replacement followed an unsigned +1 accuracy edge whose sign did not persist across runs.
  • Adopted lookup: Uniformly applying the unsigned-margin tie rule changes five cells, each losing one item, across four task types and three languages.The changed cells are distributed across math, extraction, toolcall, and abstention rather than coding.
  • Adopted lookup: 262/294 = 89.12% at $3.332910 per run is the adopted lookup result.The table leaves 12 items below the both-run oracle, including 6 coding items.
  • Against the best single model: Against Claude Opus 5, the lookup is 17 items more accurate and $4.355988 cheaper per run on the same 294 items.Opus scores 245/294 at $7.688898, while the lookup scores 262/294 at $3.332910.
  • Cheap symmetric alternative: The symmetric all-coding-Grok alternative scores 259/294 @ $0.507669, so the lookup’s 3-item accuracy edge is unresolved by this setup.The lookup costs $2.825241 more per run, a difference the procedure does resolve.

7 Limitations

The paper’s main policy and decomposition findings are constrained by in-sample selection, post-result rule choices, confounded task labels, and the limited matrix design. These limitations weaken out-of-sample, identification, and policy-selection interpretations more than the paired comparisons within the matrix.

  • 7.1 The policy was selected on the items it is evaluated on: 262/294 and the 17-item advantage over Opus are in-sample results because the lookup was fitted and evaluated on the same 294 items.The paper makes no bias correction or out-of-sample performance claim.
  • 7 Limitations: The unsigned-margin tie rule was adopted after inspecting the outcome table and choosing among three readings priced on the same data.The three readings were 267/294 at $3.368173, 262/294 at $3.332910, and 261/294 at $3.326753.
  • 7 Limitations: 87.8% of the lookup’s total cost sits in coding-ko, so refitting that signed edge could move the total more than all tie-breaks combined.The paper does not specify the direction of that possible movement.
  • 7 Limitations: Only 5 of 21 selections choose a unique both-run winner; 10 are top-scoring ties selected by cost, and 6 choose a cheaper one-behind candidate.This makes the policy total depend substantially on thin cell-level differences and tie rules.
  • 7 Limitations: The paired lookup-versus-Opus comparison remains a statement about these 294 items, but the 29-item oracle gap is not a fresh-item router gain.Both the oracle and best-single terms are hindsight maxima formed on the same data.
  • 7 Limitations: Task-type shares measure what selectors capture on seven task-and-corpus pairs, not whether task kind rather than corpus identity is the operative coordinate.Each task type draws all its items from one source, so task type and source are confounded.

7.3 Fourteen items per task×language cell

The matrix has only 14 items per task×language cell, no holdout, and several infrastructure and contamination constraints. These features limit fine-grained cell, task-type, coding, and transfer interpretations while leaving whole-matrix aggregates comparatively less affected.

  • 7.3 Fourteen items per task×language cell: 14 items per cell makes one item equal to 7.14% of a task×language cell, while the same items are used for selection and evaluation.A seven-item split would leave too little data for both fitting and estimation.
  • 7.3 Fourteen items per task×language cell: 3 infrastructure failures leave 8,229 of 8,232 keys scored, with two models evaluated on 293 rather than 294 comparable items.All failures involve the same Korean coding item.
  • 7.3 Fourteen items per task×language cell: Cell-level differences, 21 selections, per-language leaders, and per-task-type readings are weakened by the small denominators and missing cells.Whole-matrix aggregates not assembled from per-cell comparisons are less affected.
  • 7.3 Fourteen items per task×language cell: Coding uses LiveCodeBench items from 2025-01-01 through 2025-04-06 with no contamination control, and contamination could affect model differences and rankings.Coding contains much of the residual and cost, increasing the relevance of this limitation.
  • 7.3 Fourteen items per task×language cell: All seven scorers are rule-based and binary, so the task-type shares and lookup are unevaluated for open-ended writing, summarization, long-context work, and multi-turn dialogue.The study has no measurement of the production traffic mix.

7.6 One catalog snapshot, one gateway

The results are bound to fourteen models from one catalog snapshot, one gateway, and billed-cost conditions that can change across runs and over time. The language axis uses translated English-authored items, further limiting transport to native non-English traffic.

  • 7.6 One catalog snapshot, one gateway: Every headline quantity is relative to 14 models from a 2026-08-21 catalog snapshot served through one gateway.Changing the pool can alter both the best single model and the oracle gap.
  • 7.6 One catalog snapshot, one gateway: Adding or removing one model requires refitting all 21 cells, so the lookup’s selections and numbers are pool-bound.The gateway also bundles serving-side behavior into each model endpoint.
  • 7.6 One catalog snapshot, one gateway: Billed cost moves −3.81% across the matrix and −14.79% for the lookup between identical runs.The lookup’s larger movement is driven by the expensive coding-ko cell.
  • 7.6 One catalog snapshot, one gateway: Four selected models hold 17 of 21 cells, and 16 of those 17 selections were settled by cost comparisons that could change under different prices.Price changes therefore could move up to 16 of the 21 selections.
  • 7.6 One catalog snapshot, one gateway: Claude Opus 5 has no billed amount for one item, making its cost mean use 293 billed items while its accuracy denominator remains 294.The lookup-versus-Opus cost comparison therefore uses mismatched cost denominators.
  • 7.6 One catalog snapshot, one gateway: Korean, English, and Hindi share one English-authored 98-item ledger, with six task types translated by one pipeline rather than drawn from native traffic.Hindi validation affected the final instruction-item set across languages.

7.9 The noise floor is itself estimated from two runs

The paper’s noise reference is a single observed difference between two runs, with important denominator and design limitations. Model-level flip rates vary widely, and several conclusions remain sensitive to the chosen reference.

  • Two-run floor: 221 of 4,116 model-query cells flip between identical runs, a 5.37% movement that is one realization rather than an interval estimate.The paired positions would support an interval only under unverified assumptions about run-level shifts and sampling behavior.
  • Heterogeneity: 0 to 32 flips span models, corresponding to 0% to 10.88% of each model’s 294 cells.The pooled rate hides this spread, and the weighted per-model rate is approximately 5.8%.
  • Reference comparison: 3.06% of items is only a lower bound from a configuration change, not an identical-run replication.The comparison used different output caps, so the movement mixes configuration effects with run behavior.
  • Sensitivity: Nine of fourteen models lie within the 5.37% flip rate of the leader, while seven lie within the 3.06% per-item reference.The ninth-ranked model’s 4.42% margin remains unresolved under 5.37% but would be resolved below 4.42%.
  • Interpretation: The point estimates do not depend on the floor, but floor-motivated policy claims such as 262/294 and $3.332910 are partial exceptions.Those figures’ arithmetic is independent of the floor, while the policy itself was selected under a floor-motivated rule.

8 Conclusion

The conclusion carries forward the observed rerun movement and the finding that most oracle distance was visible from task type and language. It presents the static lookup as a practical result while delimiting the pilot’s scope.

  • Conclusion: 221 of 4,116 model-query cells change their correct bit across identical runs, yielding 5.37% movement and 3.81% billed-cost movement.Nothing was varied between the two executions.
  • Conclusion: 21 of the 29 oracle-improvable items are recovered by assigning one model to each task type, and language recovers 2 more.Seven of the 8 items remaining inside a task type are coding.
  • Conclusion: Five models hold the best-single-model title across ten scoring rules, and no top-two margin exceeds 8 of 294 items.Reported routing gains therefore depend on the scoring rule used for the baseline.
  • Conclusion: 262 of 294 items are answered at $3.332910 per run by the adopted lookup, versus Claude Opus 5’s 245 at $7.688898.The comparison uses an identical workload and is ahead on both accuracy and billed cost.
  • Conclusion: 23 of 29 oracle-improvable items are visible from task type and language, supporting the question of how much routing distance is already request-visible.The table was fixed in advance, but this does not show routers are unnecessary.
  • Scope and limits: The pilot covers one catalog snapshot, seven task types, three languages, one gateway, two runs, and in-sample fitting and evaluation.Binary rule-based correctness and one source per task type mean task type and corpus are the same partition.
  • Scope and limits: A third run and two sources within each task type are identified as needed before interpreting such a small residual again.These changes would provide an interval on the floor and separate task type from corpus.

A The lookup in full

The lookup is documented as an adopted task×language policy whose cell-level evidence and accuracy and cost definitions are recorded in Table 8.

  • A The lookup in full: Table 8 contains all 21 cells of the adopted task×language lookup.The table’s basis column records the evidence grade for each cell’s selection.
  • A The lookup in full: Signed evidence means a cell selection rests on an accuracy-margin sign that holds across both runs.The two unsigned grades are counted separately in §6.1.

B Scoring variants and the model pool

The appendix organizes the scoring variants, model-pool measurements, and adopted lookup used by the paper’s comparisons. It also specifies how variant distances and cell-level costs and accuracy are defined.

  • B Scoring variants: All 7 scoring variants keep the oracle above the best single model, while the best-single-model identity moves across variants.Table 9 gives each variant explicitly.
  • B The model pool: Table 10 lists each of the 14 models with both-run score, billed cost for each run, and changed correct-bit count.The model-level flip counts sum to 221 across 294 cells per model.
  • B The lookup: Table 8 reports accuracy as both-run correct out of 14 items per task×language cell and billed cost as the mean of two runs.The lookup contains all 21 task×language cells.
  • B Scoring variants: Table 9 reports distances in percentage points of items, with differing denominators when variants exclude items per model.Those distances are differences between ratios in such rows.

C Timestamps, hashes and identifiers

The paper records decision timing, hashes, and identifier controls to pin the evaluation materials and clarify what readers can verify. These records distinguish source revisions from the frozen item ledger and note that some underlying materials are not included in this version.

  • Decision records: Table 11 records each decision, its timestamp, and the passage it governs.The decision log is searched by time because a record may carry a later header time than a quoted decision.
  • Timestamps and verification: The aggregation pin is 2026-08-21T13:22Z, with USD prices from the 2026-08-21 catalog and model identifiers verified by live call on 2026-08-22.The LiveCodeBench window is 2025-01-01 to 2025-04-06.
  • Hashes: The frozen item ledger is fixed under SHA-256 903af87edfaa ..., but the ledger and label matrix are not included in this version.The hash commits to the evaluated identifiers, while §2.3 explains what readers can and cannot verify from it.
  • Hashes: Table 12 expands the abbreviated hashes printed in the body into full values for verification.Its six source-dataset revisions are distinct from the seventh, longest hash for the frozen item ledger.
Loading 2608.23023v2…