PaperlayerView exact revision on arXiv

Source-linked AI summary

RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs

Zhongzhan Huang, Guoming Ling, Yupei Lin, Yandong Chen, Shanshan Zhong, Hefeng Wu, Liang Lin

arXiv:2503.10657v2cs.CLcs.AI

TL;DR

Routing LLM research lacks comprehensive, open-source benchmarks and seeks to determine whether routing across growing model pools can scale performance. The paper analyzes more than 8,500 LLMs, introduces RouterEval with over 200,000,000 records across 12 evaluations, and finds model-level scaling alongside substantial room to improve existing routing methods. The results position Routing LLMs as a promising paradigm while identifying deployment and data-use boundaries.

  • Problem

    Existing Routing LLM methods lack comprehensive, open-source benchmarks, including sufficient candidates, diverse evaluations, and adequate performance records.

  • Method

    The paper analyzes thousands of LLM performance records and constructs RouterEval from over 8,500 LLMs across 12 evaluations, then evaluates existing router methods.

  • Results

    The paper finds that capable routers can improve performance as candidate numbers increase, even surpassing the best pool model and many strong LLMs, while existing methods retain substantial room for improvement.

  • Takeaways & Limitations

    Routing LLMs can provide model-level scaling through complementary heterogeneous candidates, including relatively weak open-source models, while RouterEval supports broader router research.

  • Takeaways & Limitations

    Deployment can become challenging with many LLM candidates, although the reported cost-effectiveness is highest at approximately 3 to 10 candidates.

Abstract

from arXiv · show

Routing large language models (LLMs) is a new paradigm that uses a router to recommend the best LLM from a pool of candidates for a given input. In this paper, our comprehensive analysis with more than 8,500 LLMs reveals a novel model-level scaling up phenomenon in Routing LLMs, i.e., a capable router can significantly enhance the performance of this paradigm as the number of candidates increases. This improvement can even surpass the performance of the best single model in the pool and many existing strong LLMs, confirming it a highly promising paradigm. However, the lack of comprehensive and open-source benchmarks for Routing LLMs has hindered the development of routers. In this paper, we introduce RouterEval, a benchmark tailored for router research, which includes over 200,000,000 performance records for 12 popular LLM evaluations across various areas such as commonsense reasoning, semantic understanding, etc., based on over 8,500 various LLMs. Using RouterEval, extensive evaluations of existing Routing LLM methods reveal that most still have significant room for improvement. See https://github.com/MilkThink-Lab/RouterEval for all data, code and tutorial.

1 Introduction

Routing LLMs assign each input to an appropriate model from a heterogeneous candidate pool to pursue objectives such as accuracy, cost, or reduced hallucination. The paper identifies model-level scaling and introduces RouterEval to address limited router benchmarks.

  • Routing LLMs: Routing LLMs use a router to select an LLM from a heterogeneous pool for each input and target goals including accuracy, computational cost, or reduced hallucination.The paradigm is compatible with models of different structures and model-enhancement methods.
  • Model-level scaling up: A capable router can rapidly improve performance as the number of candidate LLMs increases, producing a model-level scaling-up phenomenon.The paper reports that this improvement can surpass the best single candidate and many existing strong LLMs.
  • Benchmark gap: Existing Routing LLM methods lack comprehensive, open-source router benchmarks, with shortcomings including too few candidates, limited evaluation diversity, and inadequate or closed performance records.These limitations motivate the construction of RouterEval.
  • RouterEval: RouterEval contains performance records for over 8,500 LLMs and more than 200,000,000 entries across 12 evaluations covering areas such as knowledge-based question answering, commonsense inference, and semantic understanding.The benchmark is designed for router research and broad evaluation.
  • RouterEval: Evaluations using RouterEval find that existing router construction methods still have considerable room for improvement.The paper uses the benchmark to explore several methods across varied settings.

2 Preliminary

The router is formulated as a classifier that selects optimal LLMs from a candidate pool using input representations and performance-derived labels. Training may use direct records alone or additional external data, and inference produces a selection vector for unseen inputs.

  • Notation: The LLM pool is modeled as m candidate models, while each input sj is represented as κ(sj) by an encoder.The notation defines the candidate set and input representation used by the router.
  • Selection labels: For each input, a performance-derived m-dimensional one-hot vector vj marks the LLMs achieving dominant performance and indicates the router’s optimal choice.Multiple entries may be marked for tied correctness, while continuous metrics can mark models within 95% of the optimum.
  • Router training: The router rθ is trained with learnable parameters θ to predict selection vectors from input representations, using the available records and optional external data D.When no external data exists, D = ϕ.
  • Inference: After training, the router predicts a selection vector for an unseen input and uses it to choose an LLM from the pool.Router acquisition is treated as a classification problem.

3 Model-level Scaling Up Phenomenon

The paper identifies model-level scaling up in Routing LLMs: increasing the candidate pool can rapidly improve performance, particularly with capable routers, while even weak or small candidate sets can be effective.

  • Experimental Setup: The study evaluates candidate-count effects using ARC, MMMU-PRO, MATH Lvl 5, and TruthfulQA performance records from thousands of LLMs.For each candidate count, the analysis repeatedly samples candidate subsets and averages router performance.
  • Model-level Scaling Up for LLMs: As the number of candidates increases, most routers rapidly improve performance, especially when p ≥0.5.The paper interprets this as model-level scaling up, with candidate growth acting like a sparse increase in parameters.
  • Weak Candidates Can Also be Promising: Relatively weak heterogeneous candidates can complement one another, enabling fine-grained input-level division of labor and performance beyond mainstream LLMs.The paper notes that most discussed models are open-source and deployable locally, despite individually modest performance.
  • Small Number of Candidates is Enough: With only 3 ∼ 10 candidates, Routing LLMs can achieve strong performance and even surpass a strong reference model.Thus, a small candidate pool may suffice when users require only moderately good performance.

4 The Construction of RouterEval

RouterEval is constructed as a large, diverse benchmark for router research, combining performance records from thousands of LLMs across 12 evaluations with datasets and candidate groups designed for varying difficulty.

  • Benchmark Construction: RouterEval contains over 200,000,000 performance records for 12 evaluations based on more than 8,500 LLMs.The evaluations span knowledge-based Q&A, commonsense reasoning, semantic understanding, and instruction following.
  • Data Format: The router treats each input representation as an m-way selection problem, fitting a binary vector that marks optimal candidate LLMs.The dataset uses input representations κ(sj) and selection vectors vj ∈ {0, 1}m.
  • Difficulty Levels: RouterEval offers easy settings with m ∈{3, 5} and hard settings with m ∈{10, 100, 1000}.The paper focuses on easy settings because performance grows rapidly for 2 ≤m ≤10 and deployment cost is low there.
  • LLM Candidate Construction: Candidate groups are constructed as all-strong, all-weak, or strong-to-weak configurations to probe router potential from multiple perspectives.For MMLU with m = 10, an all-weak group whose individual models score below 0.3 can have oracle performance of 0.95.
  • Additional Training Data: Because direct training sets range from several hundred to tens of thousands, the benchmark also releases extra data for router training and augmentation.The released resource includes over 200,000,000 performance records for exploring augmentation, few-shot learning, regularization, pre-training, and recommendation methods.

5 Experiments

The experiments define metrics for router performance, reference-model comparison, best-single-model comparison, and classification bias, then evaluate existing routers across RouterEval settings.

  • Experimental Setting: The experiments exclude the extra data because its use is highly diverse and can be considered from multiple perspectives.Results therefore assess existing routers without that additional training resource.
  • Metrics: The original metric μo(rθ) measures the overall performance of LLMs selected by router rθ on a benchmark.The reference value VR compares router performance with a representative strong LLM such as GPT-4.
  • Metrics: The best-single-model value VB compares router performance with the strongest individual model in each candidate set.This local metric measures the router’s potential relative to its available candidates.
  • Metrics: Entropy Ep measures diversity in the router’s prediction distribution, with low entropy indicating that the router repeatedly selects the same LLM.Low entropy can signal classification bias and potential overfitting.
  • Results: Most existing routers show classification capability but lag behind the best single models and strong reference model in most settings.No single router consistently outperforms the others across benchmarks, and some routers exhibit low Ep values associated with selection bias.

6 Analysis

The analysis compares Routing LLMs with related paradigms, examines candidate-group effects and router bias, and identifies directions for improving performance. Results favor strong candidate groups and capable, debiased routers, while current methods remain limited.

  • Differences between Routing LLM and existing paradigms: Routing LLMs recommend one candidate before inference, unlike ensembles that run all candidates and aggregate their outputs.The paper frames routing as more computationally efficient while noting technical overlap with ensemble methods.
  • Differences between Routing LLM and existing paradigms: Routing LLMs can integrate heterogeneous models, whereas LLM fusion typically requires models with the same structure.The paradigm is also described as compatible with related approaches, allowing heterogeneous candidates in the pool.
  • Different candidate combinations: The "all-strong" candidate group typically performs best, while the "all-weak" group lags behind.With the capable router ro, however, the all-weak group can approach GPT-4 on MMLU even though each individual model scores no higher than 0.3.
  • Different candidate combinations: 0.3 is the maximum stated individual MMLU performance for the all-weak group, yet router ro can approximately approach GPT-4.The result indicates that heterogeneous candidates can complement one another when the router is sufficiently capable.
  • Classification bias in routers: Some routers show severe classification bias, selecting strong models with higher probability and failing to exploit complementary candidates.When Ep approaches 0, the router degenerates toward an individual router; the paper therefore identifies debiasing as crucial.
  • How to boost the performance of router: Current routers have significant room for improvement, including through performance-record data, representation learning, cold-start methods, and causal debiasing.The paper recommends focusing first on performance because adding objectives such as computational cost or hallucination rate could further compromise results with limited data.

7 Conclusion

The paper identifies model-level scaling in Routing LLMs and introduces RouterEval to address the lack of comprehensive benchmarks. RouterEval uses 200 million performance records, and evaluations show that existing routing methods still have room for improvement.

  • Conclusion: The paper identifies model-level scaling in Routing LLMs and introduces RouterEval, a benchmark built from 200 million performance records across 12 evaluations.The benchmark supports comprehensive evaluation of routing methods.
  • Conclusion: Existing routing methods still have room for improvement according to the RouterEval evaluations.This conclusion follows the paper's comprehensive exploration of the Routing LLM paradigm.

Limitations

The routing paradigm can create deployment challenges when many LLM candidates are used, while current data remains insufficient to produce an excellent router. The authors identify increasing data volume as an important future direction.

  • Many LLM candidates may create deployment challenges despite the observed model-level scaling-up phenomenon.
  • RouterEval’s current data volume still cannot produce an excellent router, partly because many benchmark performance records are closed-source and expensive.
  • Increasing the data volume is identified as an important future research direction.
  • The study covers 12 benchmarks spanning knowledge-based Q&A, commonsense reasoning, semantic understanding, mathematical reasoning, and instruction following.

B The Details of LLMs Considered

RouterEval draws on thousands of LLMs per benchmark, totaling more than 8,500 models, while filtering duplicate, erroneous, and nearly uninformative records. Most considered models are 7B open-source models with relatively weak benchmark performance.

  • Data filtering: The collected records were filtered to remove duplicates, errors, and open-source models with performance below 0.1.The paper describes such low-performing records as having almost no reference value.
  • Model coverage: More than 8,500 LLMs are included overall, with 1,800–5,000 models involved in individual benchmarks.The benchmark-specific model counts vary because different performance records use different LLMs.
  • Model characteristics: Most considered LLMs are 7B models, and most of these are relatively weak open-source models.Parameter counts are summarized in Figure 4, while performance distributions across 12 benchmarks are shown in Figure 5.
  • Implication: The main-text experiments show strong Routing LLM performance even when the candidate pool uses relatively weak models.This observation follows the statistical characterization of the candidate models.

C.1 Routing LLMs

Routing LLMs select a suitable model for each input instead of relying on a single model or invoking every candidate. RouterEval addresses the field’s lack of a unified evaluation standard while situating routing among predictive, non-predictive, ensemble, and scaling approaches.

  • Routing LLMs: Routing dynamically selects a suitable LLM for each input, reducing the need to evaluate all possible models.The paper distinguishes non-predictive methods, which invoke models sequentially, from predictive methods that select models in advance.
  • Routing methods: Predictive routers use supervised learning, reward models, performance-prediction metamodels, nearest-neighbor methods, matrix factorization, or graph neural networks.These approaches use machine-learning techniques to identify a model before inference.
  • RouterEval motivation: RouterEval provides a systematic framework intended to address the lack of a unified evaluation standard for increasingly varied routing strategies.The benchmark is presented as a foundation for evaluating router effectiveness and supporting future development.
  • Relation to ensembles: Unlike ensemble methods that generally run all candidate models and aggregate outputs, Routing LLMs assign an input to selected candidates before inference.This distinction emphasizes reduced computational overhead, although the paradigms can share optimization techniques.
  • Scaling context: Scaling-law research examines how performance relates to dataset size, model capacity, computational resources, and architectural choices.The section reviews extensions of scaling analysis across model architectures and transfer-learning settings.

F The Results on Hard Level Settings

Hard-level RouterEval evaluates routing with 10, 100, and 1,000 candidates under substantially scarcer data than the easy setting. Results are consistent with the model-level scaling-up analysis, while existing routers still leave considerable room for improvement.

  • Hard-level setting: Hard-level RouterEval uses m ∈{10, 100, 1000}, whereas the main text focuses on the easy setting with m ∈{3, 5}.The hard setting has substantially scarcer data and a more difficult classification problem.
  • Candidate scaling: Routing LLMs exhibit surprisingly strong performance when the candidate pool is typically expanded to 100–1,000 LLMs.The paper identifies a sufficient number of candidates as necessary for observing the model-level scaling-up phenomenon.
  • Results: The hard-level classification results are consistent with the model-level scaling-up analysis.Tables 6 and 7 report the classification performance of different routers.
  • Remaining gap: Existing Routing LLM methods still have considerable room for improvement on the hard-level setting.The paper presents this as the conclusion of the hard-level router comparisons.

G More Examples Visualization of Model-level Scaling Up

Across eight additional benchmarks, the authors construct routers with varying capabilities and observe that capable routing improves overall model capability as the candidate pool grows. The results also indicate that weak candidates can work together effectively even in relatively small pools.

  • Additional benchmark observations: The authors extend the model-level scaling analysis to eight additional LLM benchmarks using oracle-based routers with varying capabilities.The router construction follows the approach introduced in Section 3.
  • Router capability: As p approaches 1, the constructed router approaches the oracle router and yields the strongest classification performance among the m candidates.As p approaches 0, the router instead becomes a random sampler.
  • Model-level scaling up: Across benchmarks, a capable router increases overall model capability as the number of LLM candidates grows.This is the reported model-level scaling up pattern in the additional benchmark results.
  • Small candidate pools: Even weak candidates can achieve satisfactory performance together when the pool contains relatively few models, such as 3 to 10 LLMs.The reported result emphasizes that useful routing performance does not require a large candidate pool in every case.
  • Hard Level RouterEval: Table 7 reports Hard Level RouterEval results, distinguishing strong-router and existing-router regions and highlighting the best existing-router results.Values are rounded to two decimal places.
Loading 2503.10657v2…