Dynamic LLM Routers are Often Misguided

1 Introduction

• The paper identifies difficulty blindness, length reversal, semantic matching, and roster suboptimality as patterns prevalent across routers.It identifies length reversal as preferring to escalate queries with short answers, which tend to be easier.Source: §1 · p. 1
• Cost-accuracy Pareto efficiency on realized costs rewards difficulty blindness, length reversal, and semantic matching rather than routing by difficulty.The objective favors escalating moderately hard queries, shorter-answer queries, and routing by source identity.Source: §1 · p. 2
• The paper argues that intermediate model price points and model specialization do not empirically justify large rosters.A roster of three models, and in most cases two, closely approximates the best achievable accuracy of routers that send harder queries to stronger models.Source: §1 · p. 2
• A two-model router that avoids all four patterns offers no accuracy gain over random routing at matched cost.The paper attributes the limited gain to a well-chosen roster leaving little to route.Source: §1 · p. 2

2 Related Work

• Dynamic routers use supervised predictors of answer quality or correctness, human pairwise preferences, or semantic query clusters assigned to models.The cited examples include supervised routers (Ding et al., 2024; Lu et al., 2024; Chen et al., 2024; Feng et al., 2025; Zhuang et al., 2025; Varshney et al., 2026; Bao et al., 2026), preference routing, and semantic routers (Jitkrittum et al., 2026; Zhang et al., 2025a,b; Liu et al., 2026).Source: §2 · p. 2
• RouterBench, LLMRouterBench, RouterArena, and TwinRouterBench evaluate routers by cost-accuracy Pareto efficiency using realized generation costs.The paper argues that this objective rewards difficulty blindness, length reversal, and semantic matching.Source: §2 · p. 2
• Prior router studies report failures to beat random or dataset-level baselines, coarse routing signals, and diminishing returns from larger rosters.The paper reports that these patterns also persist in commercial routers.Source: §2 · p. 2
• The paper uses Rasch item response theory to define capability and difficulty, and its multidimensional extension to test specialization.Prior evidence suggests low- or one-dimensional LLM performance across benchmarks; the paper likewise finds specialization too weak to exploit at routing time.Source: §2 · p. 2

3 Setup

• A router assigns each query to a model in its roster; its accuracy and cost are averages over queries, with cost including input and output tokens.A two-model random router interpolates between the models’ accuracy and cost as its selection probability varies.Source: §3 · pp. 2–3
• Query difficulty is the fraction of evaluated models that fail the query, while expected answer length averages output tokens across models.The paper also fits Rasch-model capabilities and difficulties; fitted difficulty induces the same ordering as the failure fraction.Source: §3 · p. 3
• The study analyzes six commercial routers across 14 settings and independently evaluates their selected models’ generations using common settings and costs.The developers span general platforms and routing specialists.Source: §3 · p. 3
• The evaluation uses 17 benchmarks across eight categories, with 2,508 training samples and 800 equally weighted evaluation samples.The categories span coding, instruction-following, knowledge, math, question-answering, office, tool use, and Humanity’s Last Exam.Source: §3 · p. 3
• The study generates and grades responses across 26 models, using Gemini 3.7 Flash and Opus 5 (high) throughout the paper.Findings are also replicated on RouterBench, SPROUT, and LLMRouterBench using queries with graded answers for every model.Source: §3 · p. 3

4 Common Router Patterns

• The two-model random router significantly outperforms every commercial router except the vLLM Semantic Router fit on the training data.The comparison is at matched cost; the table also includes the paper’s router.Source: §4 · p. 3, Table 1 · p. 4

Difficulty blindness

• No commercial router exceeds γ = +0.20 between query difficulty and the selected model’s ability.The authors argue that the standard objective gives routers little reason to route by difficulty.Source: §4 · p. 3

Length reversal

• Answer length and difficulty correlate at γ = +0.52, but five commercial router settings have significantly negative correlation between expected answer length and selected-model mean cost.A router responding to difficulty should send longer-answer queries to stronger models.Source: §4 · p. 3

Semantic matching

• Most commercial routers send nearly all queries from a source to one model despite within-source difficulty variation; vLLM Semantic Router has the highest source AMI.Orca has essentially zero AMI because it spreads each source over dozens of models.Source: §4 · p. 4

Roster suboptimality

• Most commercial rosters include models that do not help, and Nadir omits frontier models; most routers fail the test against random routing over two well-chosen models.Nadir’s cheaper tiers contain Haiku 4.5 and Sonnet 5, neither of which is on the frontier.Source: §4 · p. 4

5 Rethinking Pareto Efficiency

• The analysis targets the standard cost–accuracy Pareto objective; Nadir's only cost-saving configurations require Haiku or Sonnet, which are outside the Pareto frontier.The authors argue that this objective encourages undesirable routing behavior.Source: §5 · pp. 4–5

5.1 Difficulty blindness

• 6.3 ± 3.0 pp: escalating the 60–80th difficulty band outperforms the hardest 20%, as stronger models add most on moderately difficult queries.Across other datasets, the same trends hold.Source: Figure 2 · p. 5, §5.1 · p. 5

5.2 Length reversal

• 59.5% accuracy at $1.03/1k queries comes from escalating the shortest 20%; escalating the 60–80th length band beats hardest-20% accuracy at 46% of the cost.The shortest-band result is 1.6 pp below hardest-20% escalation, which costs 1.5% as much.Source: §5.2 · p. 5

5.3 Semantic matching

• Source routing can beat random routing at the two cheapest budgets under realized costs, but ties or underperforms with uniform model costs; training composition can send trivial math to a strong model.Answer length tracks source closely, making source easier to predict than difficulty and the apparent advantage potentially brittle.Source: §5.3 · pp. 5–6, Figure 3 · p. 6

5.4 A revised methodology

• The revised evaluation jointly rewards hard-band escalation, easy-band accuracy, and low easy-band cost; treating hard-query escalation as desirable is an explicit, optional user-preference assumption.The authors note that partial credit could measure the distinction between models on unsolved queries, but assigning it is difficult.Source: §5.4 · p. 6

2. Length reversal: A router chooses which

• Each answer from model i is charged its mean per-query cost ci rather than its realized generation cost cij.The text motivates this choice by noting that models answer but do not determine how long answers are.Source: §2 · p. 6
• A pass-rate predictor should be evaluated by its mean γ within each data source, so assigning one pass rate per source cannot score well.Source: §2 · p. 6
• The hard band comprises queries with at least a 50% failure rate, covering 26.2% of the evaluation set.The authors do not score routers on hard-band accuracy or cost because they want difficult queries, even impossible ones, escalated to stronger models.Source: §2 · p. 6

6.1 Model granularity

• Even with perfect difficulty knowledge and evaluation-set tuning, two models are never more than 1 pp below the best roster of any size.The figure describes this as an upper-bound setting; gains measured in-sample may not generalize out of distribution.Source: Figure 4 · p. 7, §6.1 · p. 7
• In the 26-model set, Gemini 3.7 Flash trails Opus 5 (high) by 1.6 pp while costing 96% less.Source: §6.1 · p. 7

6.2 Model specialization

• They test specialization by comparing a Rasch model of model capability and query difficulty with one that adds model–query feature interactions.The Rasch model fits the outcomes well.Source: §6.2 · p. 7
• Even with evaluation-set fitting and generous feature estimates, increasing k from 0 to 8 reduces NLL only from 0.27 ± 0.02 to 0.24 ± 0.02.The authors note specialization may still matter among frontier models or older, more heterogeneous model sets; for cheap and expensive models, capability differences dominate.Source: §6.2 · p. 7

• A difficulty threshold sends queries above it to Opus 5 (high) and the rest to Gemini 3.7 Flash.The router uses SharedTrunkNet to predict difficulty from Qwen3.5-27B.Source: §7 · p. 7
• Under the standard methodology, the router’s accuracy is within noise of random routing at every budget.The roster is Gemini 3.7 Flash and Opus 5 (high).Source: §7 · p. 8
• On the easy band, both models solve nearly all queries and routed accuracies remain within noise of random; on the hard band, Opus leads by 7.2 ± 7.6 pp, also within noise.With the revised methodology, hard-band escalation surpasses random outside noise over the stated ID and OOD cost ranges.Source: §7 · pp. 7–8, Figure 5 · p. 8
• Replacing Gemini 3.7 Flash with gpt-oss-20b (low) makes both the router and difficulty oracle worse than random at every budget.The authors attribute this result to the hardest queries also tending to be the longest.Source: §7 · p. 8
• The router avoids all four patterns and distinguishes difficulty within sources better than commercial routers, but a well-chosen roster leaves little accuracy for routing to improve.Source: §7 · p. 8

• The four patterns are difficulty blindness, length reversal, semantic matching, and roster suboptimality; a two-model router avoids them but offers little measurable gain.Even with perfect difficulty knowledge, rosters larger than two gain at most 1 pp; on a well-chosen pair, the router remains within noise of random routing.Source: §8 · p. 8
• There is no reliable way to determine which queries to send to each model, and prediction may become harder for more capable models optimized for long-horizon tasks.The authors note that query text may make it difficult to predict whether future cheap models will fail a task.Source: §8 · p. 8

Limitations

• The study covers single-turn routing; per-turn myopic routing can hurt performance because switching models is not free with KV caching.The paper proposes no solution to multi-turn routing.Source: Limitations · pp. 8–9
• The datasets use English benchmark queries, may not resemble real traffic, and include only one sampled outcome per query, which can be noisy.The authors also did not test the stability of pass/fail outcomes across samples.Source: Limitations · p. 9
• Larger rosters may become useful if future models are specialized or personalized with capabilities that vary greatly by domain.Source: Limitations · p. 9
• OOD difficulty prediction within a source did not improve with tenfold more training data, suggesting the limit may be how much a query reveals about its difficulty.The authors also expect prediction could become harder as model capabilities improve.Source: Limitations · p. 9

A.1 Dataset

• The evaluation draws on benchmarks grouped into eight categories; GDPval is 3.8% of the evaluation set, and Humanity’s Last Exam uses an agent loop with web search and Python for up to 15 tool calls.Source: §A.1 · p. 13

A.2 Model set

• The model set was selected from the June 2026 frontier and revised in August 2026 with six releases; Claude Fable 5 was omitted as substantially more expensive.Source: §A.2 · p. 13

A.3 Commercial routers

• OpenRouter, Azure, Not Diamond, vLLM Semantic Router, and Nadir use popularity, estimated performance, semantic categories, or configured escalation tiers to choose models.The evaluated configurations include different cost or quality settings and two- or three-tier Nadir ladders.Source: §A.3 · p. 13
• OrcaRouter uses an adaptive LinUCB multi-armed bandit and receives 1,000 warm-up samples before evaluation.Its adaptive mode is described as balancing quality, cost, latency, and reliability.Source: §A.3 · p. 13
• Routers receive the query text as a single user message; chosen models are called through OpenRouter for consistent output grading and cost storage.For models outside the set, per-model mean cost is estimated from 100 queries stratified by source.Source: §A.3 · p. 14
• Azure scores use 797 remaining queries after three filter rejections, while Orca scores use 770 after 30 unavailable models are excluded.Seven other Orca-selected models were successfully called through the OpenAI API.Source: §A.3 · p. 14
• Several omitted routers target per-turn coding sessions, live operating conditions, hidden decisions, orchestration, or outdated model pools rather than comparable per-query routing.Nadir also requires a user-configured roster, and the tested ladders were selected to match the cost range of other routers.Source: §A.3 · pp. 14–15

A.4 Router training

• OOD difficulty predictors are fit on seven categories and tested on the held-out category, with one shared threshold despite uncalibrated scores across the eight fits.The threshold matches the stated evaluation cost using realized or per-model mean costs, but not evaluation outcomes.Source: §A.4 · p. 15

B Definitions

• Goodman-Kruskal’s gamma ranks query pairs by concordance or discordance, excluding ties; at 10% escalation, a perfect two-model router reaches Spearman’s ρ ≈0.52 but γ = 1.Source AMI measures how much source explains the router’s model choice, with 0 indicating no explanatory information in expectation and 1 indicating source fully determines the choice.Source: B Definitions · p. 15

C Router Design

• Difficulty probes use Qwen3.5-27B final-token activations, and PCA logistic probes and SharedTrunkNet are tied within noise on each metric.The authors defer to SharedTrunkNet based on prior work’s rationale that it shares information across targets, calibrates scores comparably, and computes correctness probabilities in one forward pass; the primary benefit of source exposure appears to be learning average difficulty, since within-source ordering generalizes relatively well.Source: C Router Design · pp. 15–16

F Model Ability Scores

• Model scores are matched by exact effort when available, otherwise by published or default effort, interpolation, version conversion, stand-in, or rank-only assignment.AAII scores are used because γ depends on model ordering; small score errors matter only when they change the order of models actually used by a router.Source: F Model Ability Scores · p. 25
• 55% of routed queries use an exact AA match and another 31% use a default or only published effort; score perturbations leave commercial routers’ γ at most +0.27, below the proposed router’s +0.40 OOD and +0.81 ID.The perturbation test independently varies inexactly matched scores by up to ±5 points across 1,000 draws.Source: F Model Ability Scores · p. 25
• Tables 8 and 9 list AAII scores and matching methods for the 26-model set and models selected by commercial routers, respectively.The reported entries also include model IDs and query counts for the commercial-router selections.Source: Table 8 · p. 26, Table 9 · p. 26