RouterBench: A Benchmark for Multi-LLM Routing System

Qitian Jason Hu; Jacob Bieker; Xiuyu Li; Nan Jiang; Benjamin Keigwin; Gaurav Ranganath; Kurt Keutzer; Shriyash Kaustubh Upadhyay

Source-linked AI summary

RouterBench: A Benchmark for Multi-LLM Routing System

Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, Shriyash Kaustubh Upadhyay

arXiv:2403.12031v2Mar 18, 2024cs.LGcs.AI

TL;DR

Choosing among many LLMs requires balancing performance and cost, yet router evaluation lacks a comprehensive standard. ROUTERBENCH addresses this gap with an inference-free benchmark and mathematical evaluation framework, finding substantial cost variation for comparable performance and mixed generalization across tasks.

Problem
The growing diversity of LLMs makes model selection difficult because performance and economic cost differ, while comprehensive router benchmarks are absent.
Method
ROUTERBENCH combines pre-generated responses and quality metrics with a theoretical framework for comparing routers and LLMs by inference cost and performance.
Results
2-5× monetary cost differences routinely occur for comparable performance levels, while prior routing mechanisms struggle on some complex or up-to-date tasks and simple routing performs strongly in others.
Takeaways & Limitations
ROUTERBENCH provides a standardized, scalable basis for analyzing router cost-efficiency and performance across diverse tasks.
Takeaways & Limitations
The benchmark currently evaluates only performance and economic cost, omits some models and tasks, and covers only predictive and cascading routers.

Abstract

from arXiv · show

As the range of applications for Large Language Models (LLMs) continues to grow, the demand for effective serving solutions becomes increasingly critical. Despite the versatility of LLMs, no single model can optimally address all tasks and applications, particularly when balancing performance with cost. This limitation has led to the development of LLM routing systems, which combine the strengths of various models to overcome the constraints of individual LLMs. Yet, the absence of a standardized benchmark for evaluating the performance of LLM routers hinders progress in this area. To bridge this gap, we present RouterBench, a novel evaluation framework designed to systematically assess the efficacy of LLM routing systems, along with a comprehensive dataset comprising over 405k inference outcomes from representative LLMs to support the development of routing strategies. We further propose a theoretical framework for LLM routing, and deliver a comparative analysis of various routing approaches through RouterBench, highlighting their potentials and limitations within our evaluation framework. This work not only formalizes and advances the development of LLM routing systems but also sets a standard for their assessment, paving the way for more accessible and economically viable LLM deployments. The code and data are available at https://github.com/withmartian/routerbench.

1. Introduction

LLM routing addresses the difficulty of selecting models across expanding price and performance tiers, but standardized evaluation has been lacking. ROUTERBENCH supplies a broad benchmark, theoretical comparison framework, and inference-free data for studying routers.

Motivation: LLM proliferation makes selecting an appropriate model difficult because performance and API cost vary across models and tiers.Single-model techniques can reduce serving costs but may not scale as new models emerge.
Motivation: Routing selects an LLM for each input while treating models as black boxes and balancing diverse user demands across performance and cost.This approach avoids intricate infrastructure changes and can adapt to the expanding model landscape.
Research gap: A comprehensive benchmark for evaluating routing techniques remained absent despite growing routing research and reported gains in cost efficiency and performance.ROUTERBENCH is introduced specifically to assess routers using inference dollar cost and performance.
Contributions: The construction process integrates eight datasets with eleven distinct models to develop ROUTERBENCH.The accompanying routing process dynamically allocates prompts among LLMs based on requests.
Contributions: ROUTERBENCH provides pre-generated LLM responses and quality metrics across diverse tasks, enabling router testing without additional inference.The benchmark covers major tasks and includes open-source and proprietary models.
Contributions: The benchmark introduces a theoretical framework for integrating and comparing routers and LLMs across inference cost and performance.Its mathematical formulations emphasize dollar-denominated inference cost and performance.
Results: 2-5× cost variation routinely occurs for comparable performance levels across evaluated routing strategies.The experiments assess routing efficiency across a broad range of tasks.

2. Related Work

Prior work improves individual models, synthesizes multiple outputs, or routes requests among models, but systematic router evaluation remains missing. ROUTERBENCH is presented to address that gap.

Single LLM Enhancement: Single-LLM enhancements include fine-tuning, prompting, quantization, system optimization, and mixture-of-experts methods.These approaches are generally model- and scenario-specific and do not benefit broadly from the expanding number of LLMs.
LLM Synthesis: LLM synthesis combines outputs from multiple models, but its generation-and-synthesis pipeline increases cost and latency.Strategic combinations of smaller models can match or outperform larger models, according to prior work.
Routing: Routing selects a suitable model for each input without running inference on every candidate model.Routing strategies are classified into non-predictive and predictive categories.
Routing: Non-predictive routers evaluate generated responses and select among models, often invoking models sequentially until a quality threshold is met.Related approaches also combine small models with LLMs or reroute complex queries to advanced models.
Research gap: A systematic benchmark for evaluating the many existing routers has been lacking.The paper positions ROUTERBENCH as its response to this evaluation need.

3. Math Formulation for Router Evaluation

The evaluation framework represents models and routers in a cost-quality plane, combines routing points through interpolation and convexification, and compares systems with AIQ over a shared cost domain.

Math Formulation for Router Evaluation: Router evaluation balances two conflicting objectives—maximizing efficiency and minimizing cost—through a single metric capturing this trade-off.The framework is designed to compare the multifaceted performance of routing systems.
3.1. Setup and Basic Operations: Each model output is assigned a generation cost and quality, producing expected cost and quality values across the dataset.These quantities provide the cost-quality coordinates used for model and router comparisons.
3.1. Setup and Basic Operations: A router maps a prompt and parameters to a selected LLM, with parameters potentially encoding price, latency, or router-network depth.The selected model completes the prompt as the router’s output choice.
3.1. Setup and Basic Operations: Router expected cost and performance are computed from router behavior across the dataset, while varying parameters yields points in the cost-quality plane.These points support graphical comparison with individual LLM results.
3.1. Setup and Basic Operations: Linear interpolation probabilistically mixes two routers to realize any weighted-average cost-quality point between them.The expected cost follows linearity of expectation, and the resulting point lies on the segment joining the original routers.
3.1. Setup and Basic Operations: Extrapolation extends a router’s cost domain toward infinity by adding cost without performance, and toward lower costs by interpolating with a zero-cost, zero-performance router.This permits comparison over any cost level in [0, ∞].
3.2. Non-Decreasing Convex Hull: The non-decreasing convex hull retains cost-quality combinations where higher cost does not reduce quality and replaces dominated points with superior affine combinations.For each cost in [cmin, cmax], interpolation between neighboring hull points provides optimal attainable performance.
3.2. Non-Decreasing Convex Hull: Rzero probabilistically mixes LLMs on their collective non-decreasing convex hull to maximize expected quality at a specified cost and benchmark other routers.A router is considered significant only if it outperforms Rzero.

4. Benchmark Construction - ROUTERBENCH

ROUTERBENCH combines representative datasets spanning major LLM tasks with pre-generated outcomes from diverse open-source and proprietary models. Its initial release contains 405,467 samples across 11 models, 8 datasets, and 64 tasks, enabling router evaluation without inference.

Dataset Scope: ROUTERBENCH covers commonsense reasoning, knowledge understanding, conversation, math, coding, and retrieval-augmented generation.The initial release curates eight representative datasets, including a client-query RAG dataset with manually collected ground-truth answers.
Design Principles: The benchmark is designed for broad applicability, practical industry relevance, and extensibility to metrics such as latency and throughput.Its selection principles target diverse fields and balanced task difficulty while allowing additional metrics to be integrated.
Dataset Scope: The RAG dataset contains 800 client queries across sports, history, media and art, and politics, each paired with manually collected ground-truth answers.These ground truths evaluate responses from selected LLMs and LLM-assisted search engines.
Dataset Scale: 405,467 samples cover 11 models, 8 datasets, and 64 tasks.The benchmark includes outcomes from open-source and proprietary LLMs, with model inference performed across the selected datasets.
Pilot Evaluation: Figure 3 compares all 11 LLMs by accuracy and total cost, while also showing Oracle-model selection frequencies across seven subsets.The Oracle chooses the best-performing LLM for each input and the cheapest model among ties; it achieves near-optimal performance at low cost.

5. Experiments

The experiments evaluate predictive and cascading routers across multiple tasks using cost-performance trade-offs. Predictive routers approach the best individual LLMs at similar or lower costs, while cascading routers can surpass individual models when judge errors remain low.

Predictive Router: Predictive routers estimate each model’s performance for an input and select the LLM maximizing performance minus cost.KNN and MLP predictors use a willingness-to-pay parameter to control the cost-performance trade-off.
Non-Predictive Routers: Cascading routers generate and evaluate responses sequentially, advancing through increasingly expensive models while the cumulative cost remains below T.A judge accepts the current output when its score exceeds a threshold; otherwise, the router proceeds to the next model.
Predictive Router: KNN and MLP routers reach the performance of the best individual LLMs with lower or similar costs across tasks excluding MT-Bench.The dataset is split into 70% training and 30% evaluation, while KNN and MLP configurations use tuned hyperparameters.
Predictive Router: None of the predictive routers significantly outperform the Zero router overall, although they achieve higher AIQ on MMLU and Winogrande and comparable AIQ on Hellaswag and GSM8K.They underperform the Zero router on ARC-Challenge and MBPP.
Main Results: Cascading performance improves with larger total cost budgets and lower judge error rates, with zero-error routing quickly approaching the Oracle at equal cost.With judge error rates up to 0.1, cascading routers significantly surpass individual LLMs and the Zero router on MMLU, MBPP, and GSM8K.
Main Results: Cascading performance may deteriorate rapidly when judge error exceeds 0.2, while LLM ordering remains an additional optimization factor.The reported results simulate an upper limit because the router has perfect knowledge of final scores before introducing judge error.

6. Limitations and Future Work

ROUTERBENCH is limited in evaluation criteria, model and task coverage, router designs, RAG settings, and domain-specific long-tail tasks. Future work proposes broader evaluation and expanded benchmark coverage.

ROUTERBENCH currently evaluates only performance and economic cost, omitting criteria such as latency and throughput.
Many LLMs and tasks are excluded because of time limitations, motivating future additions of newer models and broader datasets.
The study evaluates only predictive and cascading routers, leaving further router designs for investigation.
Two-stage routing with retrievers and LLMs remains unimplemented, limiting evaluation of standard RAG tasks such as HotpotQA and NaturalQuestions.
Adding domain-specific long-tail tasks, including low-resource-language translation, could expose additional aspects of LLM routing.

7. Conclusion

ROUTERBENCH provides a benchmark, dataset, and framework for evaluating multi-LLM routers through cost-efficiency and performance analysis. Its dataset records individual model outputs and evaluation attributes across diverse tasks and supports inference-free router testing.

ROUTERBENCH establishes a benchmark for standardized evaluation of routers in multi-LLM systems.
The benchmark combines a comprehensive dataset with a theoretical framework for analyzing router cost-efficiency and performance.
ROUTERBENCH records sample identifiers, model names, evaluation sources, prompts, responses, performance, costs, and true labels.
Costs use API pricing for proprietary models and Together AI reference costs for open-source models.
The benchmark uses five-shot inference for MMLU, HellaSwag, GSM8K, ARC Challenge, and Winogrande, and zero-shot inference for MBPP, MT-Bench, and RAG.
Exact match evaluates five datasets, while GPT-4 evaluates MBPP, MT-Bench, and RAG, with outcomes normalized to binary or [0, 1] scales.

A.5. Individual Dataset Result

The individual-dataset results report model and Oracle performance and cost, while Figure 7 visualizes accuracy-cost trade-offs across ROUTERBENCH sub-datasets.

The pilot study reports individual model and Oracle accuracies and costs in Table 1 across seven datasets.
Figure 7 plots each LLM’s accuracy against total cost for every ROUTERBENCH sub-dataset.

B. Extended Experimental Settings

The extended settings describe KNN and MLP router configurations and document model refusal behaviors. These behaviors include insufficient context perception and uncertainty avoidance, with further work needed on controllable output quality.

KNN routers: KNN routers vary the neighbor count and prompt-embedding model, use cosine similarity, and test 5, 10, or 40 neighbors.
MLP routers: MLP routers use one or two 100-neuron hidden layers with ReLU activation and a constant learning rate of 0.001.
Model refusal behaviors: Some models refuse certain inputs because they perceive provided context as insufficient or avoid answering when uncertain.
Model refusal behaviors: Claude 2 exhibits uncertainty-avoidance behavior most frequently among the discussed models.
Future research: The paper identifies controllable and structured output generation with automatic quality optimization as an area for future routing research.

D. Full Cascading Routers Results

This section presents the remaining cascading-router results on ARC-Challenge, MT-Bench, and HellaSwag.

The remaining cascading-router results cover ARC-Challenge, MT-Bench, and HellaSwag.
ARC-Challenge, MT-Bench, and HellaSwag are the evaluated tasks in these additional results.
The section extends the reported results for cascading routers across three benchmarks.

E. Training Data Distribution

The paper evaluates router performance and cost across in-domain and out-of-domain task settings, using multiple benchmarks and router types.

Out-domain experiments train on held-out RouterBench tasks for each dataset and evaluate on MT-Bench, MBPP, and GSM8K.
Figure 8 compares total cost and performance for eleven models and cascading routers on ARC-Challenge, MT-Bench, and HellaSwag.Different error rates are tested; AIQ is computed for Zero Router and the zero-error-rate cascading router.
Figure 9 compares total cost and performance for eleven models with KNN and MLP routers on MT-Bench, MBPP, and GSM8K.NDCH denotes the non-decreasing convex hull.