PaperlayerView exact revision on arXiv

Source-linked AI summary

When Routing Collapses: On the Degenerate Convergence of LLM Routers

Guannan Lai, Han-Jia Ye

arXiv:2602.03478v1cs.AI

TL;DR

LLM routing aims to reduce inference costs, but existing routers often choose the most expensive model even when cheaper models suffice. EquiRouter predicts per-query model rankings; on RouterBench, it achieves GPT-4-level performance at 77% of the cost versus over 93% for prior routers.

  • Problem

    Existing routers increasingly select the strongest, most expensive model as budgets rise, even when cheaper models suffice, undermining routing’s cost-reduction aim.

    Source: §1 · p. 1

  • Method

    EquiRouter directly predicts per-instance model rankings and incorporates lightweight model representations while preserving parameter sharing and inference efficiency.

    Source: §1 · p. 2

  • Results

    EquiRouter has the highest overall nAUC and lowest RCI on RouterBench and MMR-Bench; at GPT-4-level performance, it uses 77% of RouterBench cost versus over 93% for prior routers.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    EquiRouter’s lower RCI and greater use of low-cost models accompany cost savings on RouterBench and MMR-Bench.

    Source: §1 · p. 2

    The analysis considers routers trained on one query distribution and deployed on unseen queries, where misgeneralization could amplify suboptimal choices, but finds distribution shift is not necessary for collapse.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

LLM routing aims to achieve a favorable quality--cost trade-off by dynamically assigning easy queries to smaller models and harder queries to stronger ones. However, across both unimodal and multimodal settings, we uncover a pervasive yet underexplored failure mode in existing routers: as the user's cost budget increases, routers systematically default to the most capable and most expensive model even when cheaper models already suffice. As a result, current routers under-utilize small models, wasting computation and monetary cost and undermining the core promise of routing; we term this phenomenon routing collapse. We attribute routing collapse to an objective--decision mismatch: many routers are trained to predict scalar performance scores, whereas routing decisions ultimately depend on discrete comparisons among candidate models. Consequently, small prediction errors can flip relative orderings and trigger suboptimal selections. To bridge this gap, we propose EquiRouter, a decision-aware router that directly learns model rankings, restoring the role of smaller models and mitigating routing collapse. On RouterBench, EquiRouter reduces cost by about 17\% at GPT-4-level performance compared to the strongest prior router. Our code is available at https://github.com/AIGNLAI/EquiRouter.

1. Introduction

Existing routers increasingly choose the most expensive model as budgets grow, even when cheaper models suffice. EquiRouter predicts model rankings directly and reduces routing costs on RouterBench and MMR-Bench.

1. Introduction

  • LLM routing seeks to reduce inference costs by assigning easy queries to smaller models and harder queries to more capable ones.Existing approaches often predict each candidate model’s performance before making cost-aware selections.Source: §1 · p. 1
  • As budgets rise, existing routers increasingly select the most powerful, expensive model despite cheaper models being sufficient, wasting computation and money across unimodal and multimodal benchmarks.On RouterBench, existing routers almost exclusively use GPT-4 as budgets grow, while the Oracle uses it for fewer than 20% of queries.Source: §1 · p. 1
  • Collapse persists when routers train and test on the same data, while small performance perturbations can flip model rankings in a small-margin regime.Injecting unbiased noise into Oracle performance labels rapidly degrades routing quality.Source: §1 · p. 2
  • EquiRouter predicts instance-wise model rankings rather than scalar scores, using lightweight model representations while retaining parameter sharing and inference efficiency.This decision-aware design aims to reduce selection flips from small prediction errors.Source: §1 · p. 2
  • EquiRouter reduces costs by 17% on RouterBench and 12% on MMR-Bench, with lower RCI and greater use of low-cost models.A separate comparison reports GPT-4-level RouterBench performance at 77% of the cost, versus over 93% for the best previous routers.Source: §1 · p. 2
  • The proposed Routing Collapse Index (RCI) measures dominated choices and missed opportunities to select cheaper models with comparable performance.It addresses evaluation protocols that do not explicitly reveal overuse of the strongest model.Source: §1 · p. 2

2. Related Work

Prior LLM routing work includes lightweight heuristics, supervised performance predictors, and varied multi-model selection approaches. RouterBench and MMR-Bench provide evaluation settings spanning language-only and multimodal routing.

2. Related Work

  • Multi-LLM routing aims to reduce inference costs while maintaining task performance, and prior approaches predict performance, utility, or routing decisions from inputs.The cited work reports practical gains across benchmarks and deployment scenarios.Source: §2 · p. 2
  • Early heuristic routers use confidence estimates or benchmark-derived rules to choose between strong and weak models, but may not flexibly extend to diverse model pools.These methods largely focus on two-model settings.Source: §2 · p. 2
  • In two-model supervised routing, routers can estimate each model’s expected query performance and then make a cost-aware selection.This is one common formulation of routing as supervised learning.Source: §2 · p. 2
  • Multi-model approaches include kNN, MLP, linear regression, clustering, graph neural networks, dual contrastive learning, and direct decisions by open-source LLMs.Other work also estimates causal effects on task performance to select models.Source: §2 · p. 2
  • RouterBench is a widely used unimodal benchmark, while MMR-Bench extends routing evaluation to vision and language tasks.The paper’s empirical analysis primarily uses RouterBench and MMR-Bench.Source: §2 · p. 3

3. Routing Collapse: Characterization and Diagnosis

The section defines routing collapse as budget-driven convergence to the strongest model and diagnoses it as decision instability among near-tied feasible models, not a failure requiring distribution shift. The analysis motivates objectives aligned with the model rankings used at routing time.

3.1. Problem Definition

  • The Oracle selects the highest-performing budget-feasible model, breaking ties by cost; learned routers approximate it because per-query outcomes are costly to obtain directly.Source: §3.1 · p. 3

3.2. Degenerate Convergence in LLM Routing

  • As budgets grow, routers approach nearly 100% strongest-model selection on RouterBench versus under 20% for the Oracle, with similar collapse on MMR-Bench.On MMR-Bench, some routers nearly always invoke the strongest model, while others send more than 80% of queries to the two strongest models.Source: §3.2 · pp. 3–4
  • Many queries can be solved by small or medium models, yet practical routers favor GPT-4 while the Oracle still assigns a non-negligible share to WizardLM-13B.This gap indicates that collapse reflects suboptimal decisions relative to ground truth, rather than an inherent benchmark property.Source: §3.2 · p. 4

3.3. Diagnosing Routing Collapse: From Generalization to Decision Sensitivity

  • Even when trained and evaluated on the same data, routers still concentrate selections on the strongest model, so held-out generalization error is not necessary for collapse.Source: §3.3 · p. 4
  • At the largest RouterBench budget, 94.90% of queries have near-zero feasible-set margins, making small score errors sufficient to flip the top-ranked model.When near-ties dominate, these flips become frequent; as stronger models enter the feasible set with larger budgets, selections concentrate on the strongest model.Source: §3.3 · p. 5
  • Adding unbiased noise to Oracle performance labels increasingly concentrates selections on the strongest model, supporting an objective–decision mismatch between scalar-score prediction and ranking-based routing.The analysis concludes that ranking-aligned objectives are needed to address instability in the small-margin regime.Source: §3.3 · p. 5

4. EquiRouter: Paying Attention to All Models

EquiRouter directly supervises per-query model rankings to align router training with comparison-based routing decisions. It builds query-conditioned representations for each candidate and selects the highest-ranked model that satisfies the budget.

4. EquiRouter: Paying Attention to All Models

  • EquiRouter directly supervises per-query rankings, aligning its objective with routing comparisons and involving every candidate model in training.The paper presents this as a way to prevent smaller models from being ignored and improve robustness in the small-margin regime associated with routing collapse.Source: §4 · p. 5
  • At inference, EquiRouter filters out models whose predicted costs exceed the budget and selects the highest-ranked remaining model.Source: §4 · p. 5

4.1. Query and Model Representations

  • Learned model embeddings and a fixed query encoder provide the basic inputs for comparing candidate models.The query encoder may be a lightweight sentence encoder or a stronger embedding model, depending on the deployment setting.Source: §4.1 · p. 5

4.2. Model-Conditioned Model Representations

  • FiLM-style modulation conditions shared query features on each model, while joint features encode compatibility and relative offsets.The conditioning adds O(D) parameters per model embedding and is described as incurring negligible compute overhead.Source: §4.2 · p. 6

4.3. Scoring and Training Objective

  • A pairwise ranking loss trains predicted scores to preserve per-query orderings, prioritizing higher performance and lower cost when performance ties.The full objective adds standard regularization to the expected ranking loss.Source: §4.3 · pp. 6–7

5. Experiment

Experiments compare routing methods on text-only RouterBench and multimodal MMR-Bench, using curve-level and query-level metrics to evaluate cost, quality, and routing collapse. EquiRouter achieves the strongest overall trade-offs, with ablations highlighting the roles of ranking loss and query-conditioned model representations.

5.1. Experimental Setup

  • Experiments compare RouterBench and MMR-Bench using 3:1:6 train-validation-test splits and baselines spanning embedding, graph, predictor, causal, and item-response approaches.Source: §5.1 · pp. 7–8
  • Curve-level nAUC, Ps, and QNC summarize efficiency, peak performance, and relative cost, while RCI targets dominated choices and missed cheaper equivalents.The authors note curve-level metrics can miss dominated individual-query decisions.Source: §5.1 · p. 7

5.2. Experimental Results

  • EquiRouter achieves the highest overall nAUC and lowest RCI on both benchmarks; QNC is 77% vs over 93% on RouterBench and 87% vs 98% on MMR-Bench.Table 1 compares routing methods on RouterBench and MMR-Bench.Source: Table 1 · p. 8, §5.2 · p. 8
  • Removing ranking loss raises RouterBench QNC from 0.773 to 0.964 and RCI from 0.691 to 0.733, with similar deterioration on MMR-Bench.The authors attribute this to ranking loss aligning training with comparison-based routing decisions.Source: §5.2 · p. 8
  • Removing query-conditioned model representations lowers RouterBench nAUC from 0.771 to 0.755 and raises QNC from 0.773 to 0.915 and RCI from 0.691 to 0.704.The authors report similar trends on MMR-Bench.Source: §5.2 · p. 8
  • On RouterBench, EquiRouter lowers average per-query cost by about $0.002, or 17–25%, while maintaining the lowest cost across performance targets.Its relative cost improvement increases as performance targets rise.Source: Figure 5 · p. 8, §5.2 · p. 8

6. Conclusion

The conclusion identifies routing collapse as a failure mode in which routers increasingly overuse the strongest model as budgets grow, and presents EquiRouter and RCI as responses. The appendix outlines supporting reproducibility, analysis, cost-prediction, metric, experimental, and complexity material.

6. Conclusion

  • The paper characterizes routing collapse, attributes it primarily to objective–decision mismatch rather than poor generalization, and proposes decision-aware routing and RCI; experiments show improved accuracy–cost trade-offs and cost reductions.Source: §6 · p. 9

Appendix

  • The appendix covers reproducibility and protocol, further collapse analysis, cost prediction, RCI details, supplementary experiments, and complexity analysis.Source: Appendix · p. 11

A. Reproducibility & Protocol

The appendix describes the benchmarks, data splits, implementation choices, and metrics used to evaluate routing. It also outlines an in-sample experiment designed to test whether collapse depends on train–test distribution shift.

A.1. Datasets and Splits

  • RouterBench covers text-only tasks and MMR-Bench covers multimodal routing; both use fixed 3:1:6 train-validation-test splits and per-query costs based on billed tokens and model prices.The cost-performance curve is approximated by sweeping 100 evenly spaced budgets.Source: §A.1 · p. 11

A.2. Baselines and Hyperparameters

  • The appendix describes learned baselines and implementation choices, including text and image encoders, training settings, and generated model descriptions for methods that require them.RouterBench query embeddings have 384 dimensions, while MMR-Bench multimodal embeddings have 1024 dimensions.Source: §A.2 · p. 11

A.3. Evaluation Metrics

  • Evaluation reports nAUC, Ps, and QNC: curve efficiency, peak performance with its cost, and the relative cost needed to match the strongest standalone model.If a router never reaches that model’s performance, QNC is reported as “/”.Source: §A.3 · p. 12

B. Further Analysis of Routing Collapse

B.1. Experiment 1: Training-set Evaluation

The training-set evaluation removes the train–test generalization gap by training and evaluating routers on the same queries. Routing collapse nevertheless persists in this favorable in-sample setting.

B.1. Experiment 1: Training-set Evaluation

  • The in-sample protocol trains routers on the full dataset and reports nAUC, Ps, and cheapest- and strongest-model call rates across budgets.Source: §B.1 · p. 12
  • Figure A1 reports Accuracy, AUC, and strongest- and cheapest-model selection rates across budgets, with routers trained and evaluated on the same dataset.Source: Figure A1 · p. 12
  • Even when trained and evaluated on the same queries, learned routers increasingly concentrate selections on the strongest model as budgets grow.The comparison includes MLPRouter, GraphRouter, and AvengersPro against the Oracle.Source: §B.1 · p. 13

B.2. Experiment 2: Margin Distribution Beyond RouterBench

Near-ties are common beyond RouterBench, and small score perturbations can flip routing choices. Across queries, those flips can shift selections toward the strongest model as budgets expand.

B.2. Experiment 2: Margin Distribution Beyond RouterBench

  • On MIXINSTRUCT, approximately 80% of queries have top-two margins within 0.05, while exact ties among candidates are common on MMR-BENCH.These results indicate that near-ties are not unique to RouterBench.Source: §B.2 · p. 13, Figure A3 · p. 13

B.3. Why Small Margins Lead to Collapse

  • Under a budget, routing selects the highest-ranked feasible model, so a small perturbation can reverse the choice between nearly tied candidates.The example uses true performances of 0.801 and 0.800, a margin of 0.001; a 0.001 prediction error flips the ordering.Source: §B.3 · p. 13
  • When small margins are prevalent, frequent flips can shift selections toward the strongest model as budgets expand; adding unbiased noise to oracle labels also degrades routing quality rapidly.The noise intervention is consistent with the proposed explanation that comparison-based decisions are sensitive when margins are small.Source: §B.3 · pp. 13–14

B.4. Why does collapse concentrate on the strongest model (instead of an intermediate one)?

Under an additive-uncertainty model, the strongest feasible model is the most likely selection, and its selection share can grow as budgets expand. The appendix also describes cost prediction and defines a per-query index for measuring routing collapse.

B.4. Why does collapse concentrate on the strongest model (instead of an intermediate one)?

  • Under i.i.d. continuous symmetric uncertainty, the strongest feasible model has the largest selection probability, strictly so when its true performance exceeds another feasible model’s.This result explains why ranking errors need not spread probability uniformly across intermediate models.Source: §B.4 · p. 14
  • As the budget expands, the feasible set increasingly includes the globally strongest model, concentrating overall call frequency on that model.With ties or near-ties, decisions are more sensitive to noise, but the strongest feasible model remains the most likely argmax.Source: §B.4 · p. 14

C. Cost Prediction

  • The appendix describes cost prediction as a required budget-enforcement input, though cost prediction itself is not the focus of the work.Learned routers estimate per-query inference costs to filter models against the user’s budget.Source: C. Cost Prediction · p. 14, §C.1 · p. 15

C.1. Cost Predictor Used in This Work

  • For each candidate model, a two-layer MLP maps the query embedding to a vector of predicted costs.The predictor is trained on the router’s training split; at inference, models with predicted costs above budget are filtered out.Source: §C.1 · pp. 14–15
  • The cost predictor is reported as robust across datasets, with errors that do not materially affect performance–cost curves.The implementation standardizes costs during training and de-standardizes predictions at inference.Source: §C.1 · p. 15

D. Routing Collapse Index

  • The Routing Collapse Index (RCI) measures how often and how severely a router over-selects expensive models, which curve-level metrics may not reveal on individual queries.Its motivation is to expose dominated selections hidden by aggregate performance–cost frontiers.Source: D. Routing Collapse Index · p. 15

D.1. Per-query dominated decisions

  • A query counts as collapse when the router chooses a non-optimal model or misses cheaper models that match the selected model’s performance.This definition includes both dominated selections and unnecessarily expensive choices among performance-optimal models.Source: §D.1 · p. 15

C.2. Why This Treatment Is Reasonable

  • The cost MLP’s test-set MSE is ∼10−9, and replacing predicted costs with oracle costs leaves routing behavior largely unchanged, so cost estimation alone does not resolve collapse.These checks support treating cost prediction as a non-bottleneck and focusing on performance learning and the objective–decision mismatch.Source: §C.2 · p. 15, Figure A4 · p. 16, Figure A5 · p. 16

E. Supplementary Experimental Results

In supplementary evaluations, EquiRouter reduces the strongest model’s call rate while increasing smaller-model use, and maintains strong performance–cost trade-offs under out-of-domain inputs.

Supplementary Experimental Results

  • EquiRouter reduces the strongest model’s call rate by roughly 50% while increasing smaller-model use across the budget sweep.The visualization reports per-model call rates and indicates more balanced routing decisions.Source: Figure A6 · p. 16, E. Supplementary Experimental Results · p. 16
  • On RouterBench and MMR-Bench out-of-domain splits, EquiRouter achieves the highest nAUC, the lowest reported QNC, and the lowest RCI.It also improves strongest-model target success rate Ps on both benchmarks, with balanced smaller-model use not coming at the expense of final performance.Source: E. Supplementary Experimental Results · p. 16

F. Complexity Analysis

EquiRouter’s per-query computation scales linearly with the number of candidate models, with shared query processing and per-model scoring.

F. Complexity Analysis

  • EquiRouter computes a shared query representation once, then forms a joint feature and applies a shared scoring head for each candidate model.The shared trunk and per-model scoring dominate the per-query time complexity, which is linear in the number of models.Source: F. Complexity Analysis · pp. 16–17
  • Beyond the fixed query encoder, memory use is O(Kdm) for model embeddings and O(D) for intermediate activations; the authors report negligible overhead relative to invoking an LLM.The authors attribute the low overhead in practice to modest K and small D.Source: F. Complexity Analysis · p. 17
Loading 2602.03478v1…