PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

Tao Feng, Fangxu Yu, Haozhen Zhang, Zhongjie Dai, Liangqi Yuan, Zijie Lei, Weizhi Zhang, Kunlun Zhu, Haodong Yue, Keyang Xuan, Ge Liu, Jiaxuan You

arXiv:2608.06867v1cs.CL

TL;DR

LLM routing is needed because heterogeneous LLMs differ in capability and cost, while existing routers lack a common basis for fair comparison. The paper introduces LLMRouter, a modular sequential-decision infrastructure with automated supervision and evaluation, and constructs xRouteBench across multiple routing scenarios. Its study finds that router performance varies by task and budget, learned routing can outperform fixed-model selection, multi-turn routing has no consistent advantage, and user-conditioned routing improves personalization when user context is modeled well.

  • Problem

    Heterogeneous LLM capabilities and costs make model selection query-dependent, while existing routers use diverse formulations and implementations that hinder fair comparison and extension.

  • Method

    LLMRouter unifies routing as a sequential decision process with five components and automates supervision construction and joint quality–cost evaluation across candidate models.

  • Results

    Router winners vary across tasks; learned routing outperforms the strongest fixed-model baseline, multi-turn routing has no consistent gain, and user-conditioned routing improves personalization when context is modeled well.

  • Takeaways & Limitations

    A shared infrastructure and benchmark support consistent development and comparison of routing policies across single-turn, multi-turn, personalized, and multimodal settings.

Abstract

from arXiv · show

No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.

1 INTRODUCTION

LLM routing addresses heterogeneous model capabilities, costs, and user preferences, but existing routers lack a unified basis for development and fair comparison. LLMRouter provides shared abstractions, automated evaluation, and a multi-scenario empirical study.

  • No single LLM is optimal across all queries and budget constraints, making routing important for cost-effective deployment and capability or preference matching.
  • Existing routers are difficult to compare because they use incompatible formalisms, interfaces, supervision, candidate pools, and experimental stacks.
  • LLMRouter characterizes routers with context encoders, model encoders, scoring functions, decision rules, and learning signals across single-turn, multi-turn, and personalized families.
  • The infrastructure includes more than 16 representative routers and supports shared data construction, training, inference, evaluation, and deployment workflows.
  • Its pipeline evaluates every router on the same queries, candidate pool, task metrics, and token-level costs, producing xRouteBench across generic, memory, vision, time-series, and personalized scenarios.
  • No single router dominates across tasks or budgets; learned routers outperform the strongest fixed-model baseline, multi-turn routing lacks consistent gains, and user-conditioned routing improves personalization when context is modeled well.

2 A UNIFIED FORMULATION OF LLM ROUTING

LLMRouter expresses diverse routing methods as a sequential decision process whose state, candidate representations, scoring, actions, and learning signals determine quality–cost trade-offs. Its automated evaluation pipeline standardizes supervision and comparison across router families.

  • 2.1 ROUTING AS A SEQUENTIAL DECISION PROCESS: Existing routers that appear incompatible can all be formulated as a sequential decision process.
  • 2.1 ROUTING AS A SEQUENTIAL DECISION PROCESS: Routing observes a state containing the query, optional user context, and interaction history, then dispatches to candidates or terminates and aggregates responses.
  • 2.1 ROUTING AS A SEQUENTIAL DECISION PROCESS: The objective combines task-specific response quality with trajectory inference cost through a performance–cost trade-off parameter.
  • 2.1 ROUTING AS A SEQUENTIAL DECISION PROCESS: Routers are defined by context and model encoders, a scoring function, a decision rule, and a learning signal that fits these components toward the routing objective.
  • 2.1 ROUTING AS A SEQUENTIAL DECISION PROCESS: Context encoders represent routing states using embeddings or other representations, while model encoders use metadata, historical profiles, learned embeddings, or verbalized descriptions.
  • 2.1 ROUTING AS A SEQUENTIAL DECISION PROCESS: Scoring functions measure state–candidate compatibility, and decision rules select, escalate, sample, or terminate; greedy selection is optimal only when λ = 0.
  • 2.1 ROUTING AS A SEQUENTIAL DECISION PROCESS: Learning signals include stored interactions, supervised correctness labels, preference comparisons, contrastive objectives, and trajectory-level reinforcement-learning rewards.
  • 2.2 AUTOMATIC EVALUATION OF LLM ROUTING: The evaluation pipeline curates queries, collects every candidate response and token count, scores and prices them, and forms a query–model matrix reused for standardized router evaluation.

3 XROUTEBENCH: A MULTI-SCENARIO BENCHMARK FOR LLM ROUTING

xRouteBench addresses limited prior coverage by evaluating routing across heterogeneous modalities, contexts, and user-conditioned scenarios under one cost-aware protocol. LLMRouter supplies the shared infrastructure for constructing and evaluating this benchmark.

  • Prior routing benchmarks mainly cover single-turn text or image question answering and often evaluate quality without inference cost or persistent user context.
  • xRouteBench closes these gaps with a unified cost-aware protocol spanning generic LLM, memory, vision, time-series, and personalized routing.
  • All tasks share a common query schema, supervision format, and evaluation protocol, while non-text assets are transformed into self-contained textual queries with optional source pointers.
  • The benchmark contains 4,767 instances across five tracks, including knowledge and commonsense question answering, mathematical reasoning, and code generation.
  • LLMRouter organizes routing data construction, implementation and training, inference, evaluation, and deployment into six supporting modules.
  • Personalized tasks tie queries to user personas and use a persona-conditioned LLM judge, making supervision preference feedback rather than pointwise correctness.

4 THE LLMRouter LIBRARY

LLMRouter organizes routing infrastructure around shared modules while leaving router-specific components as the primary extension point. The library supports unified routing, training, evaluation, inference, and deployment across router families.

  • Architecture: The infrastructure separates router-specific logic from shared data construction, training, inference, evaluation, and deployment modules.Users primarily implement the five routing components, while the surrounding infrastructure operates unchanged.
  • Data Engine: The data engine converts task lists and candidate pools into a shared query–model matrix for supervision and testing.Adding a task requires a prompt template and registered metric, while adding a candidate requires an endpoint and token price.
  • Router Library: More than 16 routers across three families share the MetaRouter interface for single and batched routing.New routers subclass MetaRouter and implement route_single or route_batch.
  • Trainer: BaseTrainer decouples learning signals from routing code through pointwise, pairwise, or trajectory-level objectives.Non-parametric routers bypass the training module.
  • Inference and Evaluation: The route and evaluation engines use common interfaces to dispatch selected models and trace performance–cost frontiers.Multi-turn policies repeat routing until termination, while evaluation sweeps the trade-off weight.
  • Deployment: LLMRouter can expose routers through an OpenAI-compatible server for messaging platforms and live single-agent or multi-agent traffic.Routing memory preserves interaction history across turns.

5 EXPERIMENTS

The experiments evaluate routers across diverse xRouteBench tracks and deployment settings using shared candidates, metrics, and performance–cost analyses. Results show task- and budget-dependent router strengths, inconsistent benefits from multi-turn routing, and gains from user conditioning.

  • Experimental Setup: xRouteBench covers Generic LLM Tasks, memory, vision, time-series, and personalized routing across eight test sets.The candidate pool contains 18 open-weight models spanning 7B to 671B parameters.
  • Experimental Setup: Routers are scored with a weighted reward α · perf − β · cost across settings from quality-only to heavily cost-weighted evaluation.Multi-round and reinforcement-learning routers use a single configuration rather than optimizing the weighted objective.
  • Main Results: No single router dominates across tasks: RouterDC leads Generic LLM Tasks, SVMRouter leads LoCoMo, and GraphRouter has the best xRouteBench average without winning consistently.The task winner varies across benchmarks.
  • Main Results: Multi-turn routing provides no consistent gain over a single routing decision across benchmarks.Additional rounds can introduce redundant information and computational overhead, while performance also depends on the Qwen2.5-3B-Instruct base model.
  • Main Results: 68.78 persona-judge accuracy makes GMTRouter the top personalized router, ahead of PersonalizedRouter at 67.86 and EloRouter at 66.40.The 0.92-point gain over PersonalizedRouter indicates that user-context modeling matters.
  • Performance–Cost Trade-offs: Router rankings shift substantially as cost weight β increases, so deployment should select routers according to the required performance–cost operating point.MLPRouter becomes the best Vision choice for every β ≥ 0.4 despite ranking near the bottom under quality-first evaluation.
  • Performance–Cost Trade-offs: Higher inference cost generally corresponds to higher performance, but learned routes dominate always selecting the largest model.The largest model has the highest cost and only mediocre performance because smaller models solve some queries it fails.
  • Deployment: In live Slack evaluation, PersonalizedRouter reaches 83.05 agreement with held-out user preferences.The simulated ranking did not fully transfer: GMTRouter, the persona-judge winner, ranked sixth with real users.

6 RELATED WORK

Prior work spans heterogeneous router formulations and scenario-specific benchmarks. LLMRouter reframes these approaches as instances of one sequential decision process evaluated through a shared performance–cost interface.

  • LLM Routing: Single-turn routers vary in query encoding, candidate scoring, classification, reward-guided ranking, preference modeling, and query–model matching.These approaches differ substantially in their formulation and implementation.
  • LLM Routing: Personalized routers add user context and learn from preference feedback, but prior systems are typically developed and evaluated on separate stacks.LLMRouter places single-turn, multi-turn, and personalized routers behind one interface.
  • Routing Benchmarks and Evaluation: Existing routing benchmarks target distinct scenarios such as fixed-pool text or image question answering rather than a unified multi-scenario evaluation.The related benchmarks include precomputed candidate responses, aggregated performance records, preference supervision, and vision–language pools.

7 CONCLUSION

The paper presents LLMRouter as a common foundation for developing, evaluating, and deploying diverse LLM routing policies. Its shared infrastructure combines unified routing abstractions, automated evaluation, a multi-scenario benchmark, and an extensible router library.

  • Conclusion: LLMRouter casts single-turn, multi-turn, and personalized routing as instances of a common sequential decision process.The framework is intended to unify these routing settings.
  • Conclusion: The system provides automated supervision and evaluation pipelines, xRouteBench, and an open-source library implementing more than 16 routers behind one interface.It also supports deployment to real users and multi-agent systems.

A BENCHMARK DETAILS

xRouteBench contains eight test sets spanning multiple routing tracks and evaluation objectives. Its Generic LLM Tasks track is further divided into 13 subtasks.

  • xRouteBench comprises eight test sets totaling 4,767 test queries.
  • The benchmark reports exact match, multiple-choice accuracy, token-level F1, code pass rate, math answer matching, and persona-conditioned judge scores.
  • The Generic LLM Tasks track contains 13 subtasks totaling 3,729 examples.

A.1 GENERIC LLM TASKS

The Generic LLM Tasks track tests whether routers distinguish among heterogeneous text tasks with different capability requirements and output constraints. Its standardized queries preserve task-specific instructions and are scored with native benchmark objectives.

  • The track combines knowledge, commonsense, reading comprehension, mathematics, and code-generation subtasks behind one routing interface.
  • The routing question is whether query representations reveal which candidate model suits each task’s distinct capabilities and output constraints.
  • A shared text-query format places task-specific system instructions, the user problem, and any options or supporting materials into each example.
  • Responses are scored using each source benchmark’s native objective, including choice accuracy and task-specific answer evaluation.
  • Memory track: The Memory track holds retrieved evidence fixed while testing which model can use facts separated across conversational turns or sessions.
  • TimeSeries track: The TimeSeries track converts charts and raw values into common text queries, covering structural reasoning such as anomalies, periodicity, trends, and causality.

A.4 VISUAL MATHEMATICAL REASONING

The benchmark extends routing beyond text through image, video, and personalized-dialogue settings. These tasks standardize multimodal or user-conditioned inputs while preserving differences in available evidence and user utility.

  • Visual Mathematical Reasoning: Visual Mathematical Reasoning uses Geometry3K and MathVista problems whose diagrams or scientific figures contain information needed for answering.
  • Visual Mathematical Reasoning: A frozen Gemma-3-27B-IT captioner renders each image into text while withholding the solution, so all candidates receive the same image-derived query.
  • Video track: The video setting describes egocentric and exocentric views, then tests routing when one or both viewpoints are available.
  • Personalized track: The Personalized track preserves multi-turn dialogue and uses persona-conditioned comparisons to make routing targets reflect user preferences rather than universal answer quality.
  • Router details: LLMRouter routers differ in the information exposed to decisions, including queries, routing outcomes, intermediate calls, users, and interaction histories.
  • Single-turn routers: Fixed baselines always select the smallest or largest declared-parameter model, ignoring query-specific strengths.
  • Single-turn routers: EloRouter estimates a global ranking from pairwise outcomes, whereas kNNRouter, SVMRouter, and MLPRouter use query embeddings for query-specific selection.
  • Single-turn routers: RouterDC, GraphRouter, and CausalLM Router learn or predict query–model compatibility through contrastive, graph-based, or language-model approaches.

C CANDIDATE POOL AND PRICING

The candidate pool contains 18 LLMs with separately priced input and output tokens, spanning a broad cost range. Human preference data uses randomized pairwise comparisons from a ten-candidate pool.

  • The candidate pool contains 18 models whose input and output prices are reported separately in USD per 1M tokens.
  • Input prices span roughly 25×, from $0.05 to $1.25 per 1M tokens.
  • The no-cogito pool contains 17 models after removing the most expensive candidate.
  • Human preference collection samples two models from a ten-candidate pool, anonymizes and shuffles their responses, and records better, worse, or tied judgments.

E MULTI-AGENT TOPOLOGIES

The paper evaluates multi-agent routing by replacing each agent node with a router that selects a model for that call, using several coordination topologies and multi-round routing templates.

  • Topologies: Five coordination topologies structure the multi-agent experiments: Star, Tree, Graph, Chain, and Plan-Exec-Sum.Star, Tree, Graph, and Chain follow MultiAgentBench, while Plan-Exec-Sum follows GraphPlanner with width 3 and depth 1.
  • Topologies: Plan-Exec-Sum uses three atomic sub-queries whose executor answers are merged by a dedicated summarizer, unlike Star’s free-form decomposition and self-consolidation.The two topologies therefore differ in both task decomposition and answer aggregation.
  • Per-agent routing: Each functional node is replaced by a router that receives the node prompt and selects a candidate model for that call.The node prompts differ across planning, execution, and verification, testing whether routers trained on ordinary queries generalize to multi-agent settings.
  • Multi-round routing: kNN-MultiRound and LLM-MultiRound decompose queries, obtain sub-answers, and aggregate them, differing in nearest-neighbor versus LLM-based model selection.Their shared multi-round structure separates query decomposition, sub-query routing, and final aggregation.
  • Multi-round routing: Router-R1 can request specialist models repeatedly through search tags, inserting returned responses as information before producing the final answer.It requires exact candidate model names and ends with an answer tag when no further external knowledge is needed.
  • Multi-round routing: AutoMix decomposes a query into one to four meaningful sub-queries and uses a verifier to judge generated answers as correct or incorrect.Its prompts also instruct the system to maximize distinct, nonredundant sub-queries and output only the decomposed queries.

G THE COMFYUI VISUAL INTERFACE

The ComfyUI interface represents LLMRouter as an editable graph connecting data construction, routing, and evaluation. Nodes expose datasets, candidate models, router families, and configuration controls within one pipeline.

  • Pipeline graph: The ComfyUI graph connects source benchmarks and candidate models to a data-engine node that emits the query–model matrix, which router nodes consume for evaluation.The matrix records each candidate’s per-query performance and cost, providing the supervision used by routers.
  • Pipeline graph: Every graph node is a library component and every edge is an artifact flowing between components, making the routing pipeline visible and editable.Replacing a router changes one node while the data-engine and evaluation nodes remain in place.
  • Data construction: The data nodes declare datasets and LLMs, while Generate Data performs query curation, response collection, scoring, and matrix writing in one call.The resulting data directory is self-contained, and router nodes use defaults from the shared YAML configuration.
  • Router configuration: Router nodes are grouped by the three router families and expose library hyperparameters through typed widgets.The menu organization makes the paper’s router taxonomy directly visible in the interface.
Loading 2608.06867v1…