RouteLLM: Learning to Route LLMs with Preference Data

1 INTRODUCTION

• Model heterogeneity creates a deployment trade-off: larger models generally offer higher quality at higher cost, while cheaper models can struggle with complex queries.Source: §1 · p. 1
• An ideal router invokes one LLM per query to limit cost and latency, generalizes across domains, and supports different models without retraining.Source: §1 · p. 1
• Over 2x cost savings on popular benchmarks are achieved with minimal impact on response quality.Evaluations include MMLU and MT Bench.Source: §1 · p. 2
• The framework uses human preference data and data augmentation to learn routers that choose between strong, expensive models and weak, cheaper models.Routing minimizes cost at a specified performance target by assigning simpler queries to the weak model and reserving complex queries for the strong model.Source: §1 · p. 2
• A single trained router generalizes to unseen data and maintains strong performance across multiple strong/weak model pairs without retraining.Source: §1 · p. 2
• The authors open source a framework for training, serving, and evaluating routers, enabling users to train their own routers and compare benchmark performance.Source: §1 · p. 2

2 RELATED WORK

• Reward modeling (Ouyang et al., 2022) assesses generated responses, whereas routing selects an LLM beforehand using query complexity and available models’ capabilities.Routing therefore requires understanding both the query and the capabilities of candidate models before a response exists.Source: §2 · p. 2
• RouteLLM routes each query to one LLM, unlike multi-query approaches that can increase latency.LLM-Blender (Jiang et al., 2023) queries multiple models and selects the best response; Frugal-GPT (Chen et al., 2023) sequentially queries models until obtaining a reliable response. AutoMix (Aggarwal et al., 2024) uses smaller-model self-verification before potentially escalating to a larger model.Source: §2 · p. 2
• Unlike Hybrid-LLM (Ding et al., 2024), RouteLLM uses human preferences, explores multiple router architectures, and reports significant augmentation gains across architectures.Hybrid-LLM uses synthetic MixInstruct (Jiang et al., 2023) preferences based on BARTScore (Yuan et al., 2021) and one BERT-based router; RouteLLM uses Chatbot Arena human preferences. Hybrid-LLM evaluates on the MixInstruct test split without evidence of out-of-domain generalization, whereas RouteLLM evaluates on several decontaminated public benchmarks to examine this.Source: §2 · pp. 2–3
• Unlike Zooter (Lu et al., 2023), RouteLLM mainly uses human preferences and maintains strong performance with LLMs absent from training.Zooter uses labels from QwenRM reward models (Bai et al., 2023), which can inherit training-data biases affecting routing reliability. It explores only a BERT-style router, and its training signal depends on a fixed LLM set, limiting adaptability.Source: §2 · p. 3

3.1 PROBLEM FORMULATION

• A preference-trained binary router chooses between costly, higher-quality strong models and cheaper, lower-quality weak models using predicted strong-model win probabilities.Training comparisons record strong-model wins, ties, or weak-model wins; win-prediction parameters are learned by maximizing the likelihood of observed preferences.Source: §3.1 · p. 3
• The cost threshold α selects the strong model when its predicted win probability is at least α, and otherwise selects the weak model.α lies in [0, 1]; increasing it favors cheaper weak models, while decreasing it favors higher-quality, more expensive strong models.Source: §3.1 · p. 3

3.2 METRICS

• Basic evaluation measures cost efficiency through the percentage of calls to strong models and performance through average response quality.Strong-model call percentage serves as the cost measure because strong models are substantially more expensive than weak models.Source: §3.2 · p. 4
• Performance gap recovered (PGR) measures recovery of the strong–weak performance gap; average performance gap recovered (APGR) summarizes recovery across cost constraints.APGR uses a call-performance curve generated by varying α. Its empirical approximation evaluates ten strong-model call percentages within [0%, 100%], selecting a threshold for each cost constraint.Source: §3.2 · p. 4
• Call-performance threshold CPT(x%) is the minimum percentage of strong-model calls required to achieve a performance gap recovered of x%.This metric expresses the cost required to reach a desired performance level.Source: §3.2 · p. 4

4.1 CHATBOT ARENA DATA

• 80k Chatbot Arena battles supply human preferences for router training, retaining queries and winner identities but excluding model responses.Users select winners or ties, but comparison labels between any two models average less than 0.1%. Dynamic programming groups models into 10 leaderboard-based tiers minimizing within-tier variation; the top two represent strong models and the third represents weak models.Source: §4.1 · pp. 4–5

4.1.1 DATA AUGMENTATION

• Approximately 1500 MMLU validation questions (Hendrycks et al., 2020) provide additional comparison labels by checking strong- and weak-model responses against known correct answers.This golden-labeled augmentation uses the benchmark’s multiple-choice questions.Source: §4.1.1 · p. 5
• Approximately 120K GPT-4-judged preference samples cost around $700 USD, using Nectar queries with existing GPT-4 responses and generated Mixtral-8x7B responses.GPT-4 serves as the strong model and Mixtral-8x7B as the weak model, with existing responses reducing collection costs. Augmentation addresses cross-class preference sparsity that persists after tiering and can limit generalization, particularly for parameter-heavy routers.Source: §4.1.1 · p. 5

4.2 ROUTING APPROACHES

• Similarity-weighted ranking solves a query-weighted Bradley-Terry model at inference, while matrix factorization learns bilinear model–query scores from preferences.Ranking weights training queries using scaled cosine similarity and requires no prior training. Matrix factorization learns model and query embeddings, modeling win probabilities through a sigmoid of score differences.Source: §4.2 · pp. 5–6
• The BERT classifier fully fine-tunes BERTBASE (Devlin et al., 2018), predicting win probability through a sigmoid logistic-regression head on the query’s CLS embedding.Training uses approximately 2000 steps on 2xL4 24GB GPUs, with batch size 16 and maximum sequence length 512.Source: §4.2 · p. 6
• The causal classifier uses Llama 3 8B (AI@Meta, 2024b), predicting comparison-label tokens rather than using a separate classification head.An instruction prompt contains the user query; comparison labels are added to the vocabulary, and a softmax over label classes yields win probabilities.Source: §4.2 · p. 6

5 EXPERIMENTS

• 65k Arena pairwise comparisons across 64 models, grouped into 10 classes, support router training with optional golden-label or GPT-4-judge augmentation.Training reserves 5k validation samples and removes prompts shorter than 16 characters; golden labels come from the MMLU validation split. Decontaminated evaluations on MMLU (Hendrycks et al., 2020), MT Bench (Zheng et al., 2023), and GSM8K (Cobbe et al., 2021) assess out-of-domain generalization, routing between gpt-4-1106-preview (OpenAI, 2023) and Mixtral 8x7B (Jiang et al., 2024) against cost-constrained random routing.Source: §5 · pp. 6–7

5.1 RESULTS

• 50% fewer GPT-4 calls than random are required by the best judge-augmented MT Bench router, matrix factorization.Arena-only matrix factorization and similarity-weighted ranking outperform random, whereas BERT and causal classifiers remain near random; judge augmentation improves all routers, including over 50% higher BERT APGR than random. The authors also report substantial improvements over commercial routing systems in Appendix E.Source: Table 1 · p. 7, §5.1 · p. 7
• Approximately 20% fewer GPT-4 calls than random achieve CPT(50%) across all MMLU routers after augmentation with approximately 1,500 golden-labeled samples.These additional samples constitute less than 2% of training data. Arena-only routers perform near random, which the authors attribute to MMLU questions being out-of-distribution.Source: Table 2 · p. 7, §5.1 · p. 8
• 17% fewer GPT-4 calls than random achieve CPT(50%) and CPT(80%) with the judge-augmented causal LLM classifier, the best GSM8K router.Arena-only routers perform close to random. Judge-labeled augmentation raises every router’s APGR from below random to above random.Source: Table 3 · p. 8, §5.1 · p. 8

5.2 ADAPTABILITY ACROSS MODELS

• Routers retain comparable MT Bench performance without retraining when switched to unseen Claude 3 Opus/Sonnet and Llama 3.1 70B/8B model pairs.The same Arena-plus-judge-trained routers significantly outperform random on both replacement pairs. The authors report approximately half the random router’s GPT-4 calls at CPT(80%), using that terminology despite replacing GPT-4.Source: Table 4 · p. 8, §5.2 · pp. 8–9

5.3 QUANTIFYING DATASET AND BENCHMARK SIMILARITY

• Higher benchmark–dataset similarity correlates with stronger routing performance, while augmentation increases similarity by aligning preference-data distributions with evaluation benchmarks.The authors attribute stronger Arena-trained performance on MT Bench than MMLU or GSM8K to greater distributional similarity. They propose similarity scores as a guide to improving routers for known query distributions.Source: §5.3 · p. 9, Table 5 · p. 9

5.4 COST ANALYSIS

• Up to 3.66x cost savings are reported while maintaining response quality, using GPT-4-call ratios relative to random routing.The analysis estimates GPT-4 and Mixtral 8x7B costs at $24.7 and $0.24 per million tokens, respectively, treating GPT-4 calls as the dominant cost.Source: Table 6 · p. 9, §5.4 · p. 9

5.5 ROUTING OVERHEAD

• 0.4% of GPT-4 generation cost bounds the additional cost of similarity-weighted ranking, the most expensive router evaluated.Overhead estimates combine profiled request throughput, VM hourly prices, and embedding-generation costs where applicable. GPU routers on an NVIDIA L4 are more efficient than CPU-based similarity-weighted ranking.Source: §5.5 · p. 10, Table 7 · p. 10

6 CONCLUSION

• The routers achieve significant cost savings and high response quality across diverse benchmarks without excessive cost or latency overhead.Evaluations span open-ended question answering, humanities, and math problems.Source: §6 · p. 10
• The routers maintain performance across multiple strong/weak model pairs without retraining.The authors identify this transfer capability as important to practical usefulness.Source: §6 · p. 10
• Augmenting training data with an LLM judge or in-domain data enables routers to outperform the random baseline across all benchmarks.Training solely on Darena performs poorly on MMLU and GSM8K. Gains are greatest when training and evaluation data closely resemble one another, as indicated by the benchmark-dataset similarity score.Source: §6 · p. 10
• Real-world query distributions may differ substantially from evaluation benchmarks, but collecting a small amount of in-domain data can improve application-specific routing through augmentation.Source: §6 · p. 10
• Router selection should consider latency, cost, and query types rather than assume a single best router.Routers trained on the same dataset can perform very differently on the same benchmark; explaining this variation remains future work.Source: §6 · p. 10
• The study focuses on two-model routing, leaving extension to multiple models as a promising future direction.Source: §6 · p. 10

• Evaluation examples flagged as contaminated are discarded before reporting uncontaminated scores.Source: B DATA CONTAMINATION · p. 14
• 0.95 is the similarity threshold used to search all training data for matches to each evaluation example.Source: B DATA CONTAMINATION · p. 14
• OpenAI’s text-embedding-3-small model generates embeddings for both evaluation data and preference training data.Source: B DATA CONTAMINATION · p. 14

C BENCHMARK-DATASET SIMILARITY

• The benchmark–dataset similarity score averages each evaluation prompt’s maximum similarity across all preference-dataset samples.Each evaluation prompt contributes its closest-match similarity rather than an average over preference samples.Source: C BENCHMARK-DATASET SIMILARITY · p. 14
• The score compares embeddings of benchmark prompts with embeddings from a specific preference dataset, allowing different numbers of evaluation and preference samples.Source: C BENCHMARK-DATASET SIMILARITY · p. 14
• The maximum-similarity choice assumes that a few highly similar preference samples are more valuable for efficient query routing than many less-similar samples.Source: C BENCHMARK-DATASET SIMILARITY · p. 14

D COST CALCULATION

• $10 per 1 million input tokens and $30 per 1 million output tokens are the gpt-4-1106 endpoint prices used in the analysis.This endpoint was used for the evaluations.Source: D COST CALCULATION · p. 14
• 95 input tokens and 264 output tokens are the training-set averages used to weight costs under a short-prompt, single-turn assumption.These lengths imply an input-to-output token ratio of roughly 95/264.Source: D COST CALCULATION · p. 14
• 24.7 USD per 1 million tokens is the estimated GPT-4 average cost, compared with $0.24 USD per 1 million tokens for Mixtral 8x7B.The Mixtral estimate assumes equal input and output token prices.Source: D COST CALCULATION · pp. 14–15

E INDEPENDENT BENCHMARKS

• Up to 40% fewer GPT-4 calls achieve the same MT Bench performance as commercial systems using matrix factorization and causal LLM routers trained on Darena +Djudge.The commercial comparisons are Unify AI and Martian.Source: Figure 2 · p. 15, E INDEPENDENT BENCHMARKS · p. 15
• Routers pair gpt-4-turbo-2024-04-09 (OpenAI, 2023) with mixtral-8x7b-instruct-v0.1 (Jiang et al., 2024) or llama-2-70b-chat (Touvron et al., 2023), depending on system support.GPT-4 serves as the strong model; Mixtral or Llama serves as the weak model.Source: E INDEPENDENT BENCHMARKS · p. 15
• Commercial baselines use Unify AI’s best-performing dashboard configuration and Martian’s performance-optimized setting.Martian’s maximum cost is $10.45 per million tokens, targeting 50% of calls routed to GPT-4. This approximates public inference costs (OpenAI, 2024; Together.AI, 2024) assuming a 1:1 input:output token ratio.Source: E INDEPENDENT BENCHMARKS · p. 15

F ADDITIONAL PLOTS

• MT Bench performance is plotted for all routers.Figure 3 supplements the results presented in Section 5.1.Source: F ADDITIONAL PLOTS · p. 15, Figure 3 · p. 16
• 5-shot MMLU performance is plotted for all routers.Figure 4 covers this evaluation setting.Source: Figure 4 · p. 16
• 8-shot GSM8K performance for all routers appears among the additional plots supplementing Section 5.1.Figure 5 covers this evaluation setting.Source: F ADDITIONAL PLOTS · p. 15, Figure 5 · p. 16