Source-linked AI summary
LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing
Hao Li, Yiqun Zhang, Zhaoyan Guo, Chenxu Wang, Shengji Tang, Qiaosheng Zhang, Yang Chen, Biqing Qi, Peng Ye, Lei Bai, Zhen Wang, Shuyue Hu
TL;DR
LLM routing seeks to assign each query to the most suitable model, but fragmented and costly evaluation has limited reliable comparison. LLMRouterBench addresses this with a large-scale unified benchmark and framework, finding strong model complementarity alongside similar router performance, failures against Oracle, and diminishing returns from larger ensembles.
Problem
Routing research requires costly multi-model evaluation, while fragmented pipelines and protocols hinder fair comparison, reproducibility, and cumulative progress.
Method
LLMRouterBench consolidates over 400K instances from 21 datasets and 33 models, unified metrics for two routing paradigms, and 10 representative baselines.
Results
Under unified evaluation, models remain complementary, but many routers perform similarly, several—including commercial routers—do not reliably outperform Best Single, and a substantial gap remains to Oracle.
Takeaways & Limitations
Routing progress depends on improving model recall and curating ensembles, since embeddings have limited impact and adding models yields diminishing returns.
Takeaways & Limitations
The benchmark covers recent routing methods with public implementations and excludes domain-specific, very long-context, and multimodal settings.
Abstract
from arXiv · showhide
Large language model (LLM) routing assigns each query to the most suitable model from an ensemble. We introduce LLMRouterBench, a large-scale benchmark and unified framework for LLM routing. It comprises over 400K instances from 21 datasets and 33 models. Moreover, it provides comprehensive metrics for both performance-oriented routing and performance-cost trade-off routing, and integrates 10 representative routing baselines. Using LLMRouterBench, we systematically re-evaluate the field. While confirming strong model complementarity-the central premise of LLM routing-we find that many routing methods exhibit similar performance under unified evaluation, and several recent approaches, including commercial routers, fail to reliably outperform a simple baseline. Meanwhile, a substantial gap remains to the Oracle, driven primarily by persistent model-recall failures. We further show that backbone embedding models have limited impact, that larger ensembles exhibit diminishing returns compared to careful model curation, and that the benchmark also enables latency-aware analysis. All code and data are available at https://github.com/ynulihao/LLMRouterBench.
1 Introduction
LLM routing has grown amid rising model availability, but costly and fragmented evaluation has limited reliable progress. LLMRouterBench addresses these barriers with unified large-scale evaluation, revealing strong model complementarity alongside surprisingly similar router performance and a persistent Oracle gap.
- Evaluating routing methods is costly because each method must test multiple large models across many datasets, while fragmented pipelines hinder fair comparison and reproducibility.
- LLMRouterBench provides over 400K instances from 21 datasets and 33 models, supporting performance and performance-cost routing with unified evaluation across 10 baselines.The benchmark includes recently released flagship and proprietary models, comprehensive metrics, and adapters for public routing implementations.
- Models show clear complementarity, but contemporary routing methods deliver nearly indistinguishable results across performance metrics under a unified model ensemble and dataset suite.The similar-performance pattern extends to performance-cost trade-offs, where several recent methods, including OpenRouter, do not reliably outperform Best Single.
- A substantial gap remains to the Oracle because routers frequently fail to recall the sole candidate model that answers a query correctly.The Oracle selects the best-performing model per query in hindsight.
- Embeddings have little impact on routing performance, while adding models shows diminishing returns compared with carefully curating the ensemble.The benchmark also supports latency-aware analysis for performance-cost-latency optimization.
2 Related Work
Prior work studies routing for ensemble performance and performance-cost trade-offs, using methods that select or combine models under different objectives. Existing benchmarks, however, differ in models and reporting, limiting direct comparison and algorithm development.
- Existing routing benchmarks use different models, while RouterArena reports only black-box overall performance without per-prompt or per-model details.These design differences constrain direct comparison and prevent RouterArena from directly supporting development of new routing algorithms.
- Routing for Performance: Performance routing selects models likely to answer correctly, with prior methods using pairwise ranking, dual contrastive learning, embeddings, or structured candidate selection.
- Routing for Performance-Cost Tradeoff: Performance-cost routing trades answer quality against financial or computational cost, often routing between a smaller and a larger model.
3 LLMRouterBench
LLMRouterBench is a unified benchmark and framework covering performance-oriented and performance-cost routing across diverse models and datasets. It standardizes data collection, evaluation, metrics, baselines, and Pareto analysis for reproducible comparison.
- Routing settings: LLMRouterBench supports performance routing and performance-cost tradeoff routing across two model pools: 20 lightweight models and 13 flagship models.The lightweight pool contains comparable ∼7B models, while the flagship pool varies substantially in capability and cost.
- Datasets: The benchmark covers 21 datasets spanning mathematics, code, logic, knowledge, affective tasks, instruction following, and tool use.Dataset selection differs by setting: 15 datasets for lightweight models and 10 for flagship models, excluding unsuitable datasets for meaningful comparisons.
- Framework: Its modular Collector, Evaluator, and Adaptor design standardizes model outputs, dataset-specific evaluation, and routing-algorithm integration.The Collector handles caching, retries, and cost tracking; the Evaluator applies dataset-specific metrics; and the Adaptor converts standardized data for algorithms.
- Evaluation metrics: The evaluation compares routing methods with Random Router, Best Single, and Oracle baselines, using Gain@R, Gain@B, and Gap@O as reference metrics.Random Router is a lower bound, Best Single selects the highest-average-accuracy model, and Oracle selects a correct model per instance when available.
- Performance-cost analysis: Performance-cost evaluation measures best achievable performance, cost savings at Best Single-level accuracy, and distance to the Pareto frontier.PerfGain and CostSave summarize the first two tradeoffs, while ParetoDist measures average distance from frontier configurations.
- Scale: The benchmark contains 23,945 prompts, 391,645 instances, and approximately 1.8B tokens across its two routing settings.Collection required about 1K GPU hours for lightweight inference and $2,771.84 in API costs.
4 Experiments
Experiments confirm strong complementarity across models, while showing that leading routers perform similarly and remain substantially below the Oracle. Careful model selection often matters more than simply enlarging the pool, and routing can improve performance-cost trade-offs while supporting latency-aware analysis.
- Performance-oriented setting: Models show complementary strengths across domains: mathematics, code, logic, and affective benchmarks are often led by different models.
- Performance-oriented setting: Leading routers achieve broadly comparable results across Gain@R, AvgAcc, Gain@B, and Gap@O, while Avengers relies on clustering without neural-network training.
- Performance-oriented setting: A substantial Oracle gap persists because routers often fail to recall the lone candidate model that produces a correct answer.
- Performance-oriented setting: Embedding-model substitutions produce no significant performance differences, suggesting routing mechanisms—not semantic representations—may be the more relevant bottleneck.
- Performance-oriented setting: Diminishing returns arise as more models are added, whereas a well-chosen small subset can outperform a larger random pool.
- Performance-cost setting: Models complement one another across performance, cost, and latency, with some models offering similar accuracy and cost but markedly different response times.
- Performance-cost setting: Top routers achieve up to a 4% average-accuracy gain or a 31.7% cost reduction at matched Best Single performance, but several routers fail to beat that baseline.
- Performance-cost setting: Avengers-Pro is nearly Pareto-optimal, generally offering lower cost at comparable performance or higher performance at comparable cost.
5 Conclusions
LLMRouterBench consolidates large-scale data, unified metrics, and representative baselines for evaluating LLM routing across performance and performance-cost settings. Its evaluation confirms model complementarity but finds similar router performance, persistent Oracle gaps from recall failures, limited gains from embedding choice or ensemble scaling, and opportunities for latency-aware routing.
- LLMRouterBench unifies over 400K instances spanning 21 datasets and 33 models with comprehensive metrics and 10 representative routing baselines.
- The results reaffirm model complementarity but show that most routing methods perform similarly under unified evaluation.
- Several recent approaches, including commercial routers, fail to reliably outperform a simple baseline.
- A persistent gap to the Oracle is driven primarily by systematic model-recall failures rather than insufficient ensemble capacity.
- Backbone embedding choices and aggressive ensemble-size scaling provide limited practical gains, while the benchmark also supports latency trade-off analysis.
Limitations
LLMRouterBench identifies three limitations: incomplete router coverage, restricted benchmark settings, and approximate latency estimates tied to a specific provider configuration.
- The benchmark evaluates many routing methods but does not cover all existing approaches, focusing on recent methods with publicly available implementations.Additional routers can be integrated through lightweight adapters without reimplementing the full pipeline.
- The benchmark excludes domain-specific verticals, very long-context tasks, and multimodal benchmarks because it reflects common evaluation regimes for contemporary LLMs.The routing formulations, metrics, and analysis procedures can be extended by adding new dataset evaluators.
- Latency estimates are indicative rather than definitive because they combine token usage with OpenRouter throughput figures from a specific provider configuration.The paper explicitly characterizes this latency analysis as approximate.
A Further Related Work
Prior LLM-routing benchmarks differ in model coverage, dataset difficulty, cost reporting, and evaluation consistency, limiting direct cross-method comparison.
- RouterBench targets multi-LLM routing but is restricted to early-generation models and eight relatively simple datasets.
- EmbedLLM, RouterEval, and FusionFactory benchmark routing over open-source models, while EmbedLLM and RouterEval omit inference cost information.
- RouterArena compares routing systems as black boxes but uses different model pools across routers, undermining cross-method comparability.
B.1 Data Collection
The data collection setup combines local batched inference for approximately 7B open-source models with API collection for flagship models under controlled decoding and retry procedures.
- Approximately 7B open-source models are deployed on NVIDIA A800-80G GPUs using vLLM 0.8.4 for efficient batched inference.
- Flagship model outputs are collected through OpenRouter, except GLM-4.6 and Intern-S1, which use official APIs.
- All generations use temperature 0.2 and top_p 1.0, with remaining decoding parameters set to defaults.
- API requests are retried up to 10 times upon failure, while requests exceeding that limit are marked as failures.
B.2 Experimental Setup
The experiments standardize data splits, randomization, embeddings, and binary-router model choices to support consistent baseline comparisons, while documenting dataset and model licensing.
- Experiments use a 70% training and 30% test split, repeated five times with random seeds 42, 999, 2024, 2025, and 3407.
- Embedding-dependent methods uniformly use gte-qwen2-7B-instruct embeddings.
- Binary routers route between Qwen3-235B and GPT-5, pairing the strongest model with a much cheaper competitive alternative.
- The paper states that all datasets and models are publicly available, properly cited, and used consistently with their licenses and intended research purposes.
B.3 Baselines
The benchmark evaluates routing baselines using official implementations or APIs, with a unified embedding backbone where applicable, across performance, performance–cost, inference-cost, and latency-aware analyses. It also documents the model pools, datasets, evaluation splits, and routing distributions used in these comparisons.
- Implementation: Most neural baselines replace their original encoders with gte-qwen2-7B-instruct for consistent embedding comparisons.RouterDC, GraphRouter, EmbedLLM, HybridLLM, RouteLLM, FrugalGPT, and MODEL-SAT use this embedding model, with MODEL-SAT additionally using Qwen2.5-7B-Instruct as its language model.
- Implementation: RouteLLM and HybridLLM use GPT-5 as the strong model and Qwen3-235B as the weak model, while RouteLLM applies a 0.5 win-rate threshold.FrugalGPT uses the same strong and weak model pairing and adopts its cascade strategy.
- Implementation: The evaluated baselines include RouterDC, GraphRouter, Avengers(-Pro), OpenRouter, MODEL-SAT, EmbedLLM, HybridLLM, RouteLLM, and FrugalGPT.Avengers uses kmeans clustering with k = 64 and Avengers-Pro evaluates 101 performance-coefficient configurations; GraphRouter reports Performance First, Balance, and Cost First configurations.
- Evaluation: Performance and performance–cost comparisons use 30% test splits averaged over five random seeds, with best and second-best results marked separately.The benchmark reports results for each dataset and includes explicit handling of tool-calling incompatibility on τ 2-Bench for GPT-5-Chat and OpenRouter.
- Evaluation: The evaluation reports model and routing-method performance, inference cost in $/1M tokens, dataset details, and model pools for both routing settings.The performance-oriented pool contains open-source models around 7B parameters, while the performance–cost pool contains flagship models.
- Evaluation: OpenRouter’s supported model pool is time-specific, and Figure 10 compares its routing distribution with Avengers-Pro under cost-matched and highest-accuracy settings.Models selected in less than 5% of queries are grouped into “Others”; the reported Avengers-Pro coefficients are α = 0.62 and α = 1.00.