PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey

Yasmin Moslem, John D. Kelleher

arXiv:2603.04445v2cs.NIcs.CLcs.PF

TL;DR

Static deployment struggles with queries that differ in complexity, domain, and required capability, creating a cost-performance dilemma. This survey synthesizes dynamic routing and cascading across independently trained LLMs through a taxonomy and three-dimensional framework. It concludes that strategic use of model complementarity can outperform the strongest individual models while improving efficiency, although multimodal and multidimensional routing remain open challenges.

  • Problem

    Queries differ substantially in complexity, so deploying one model for all requests can waste resources on simple queries or inadequately handle complex ones.

  • Method

    The survey analyzes multi-LLM routing and cascading paradigms and organizes systems by decision timing, information signals, and computational mechanisms.

  • Results

    Effective routing systems can outperform even the strongest individual models by leveraging model complementarity and specialization while achieving considerable efficiency gains.

  • Takeaways & Limitations

    Choosing a routing strategy requires balancing competing objectives under deployment and computational constraints.

  • Takeaways & Limitations

    Multimodal routing remains underexplored, with unresolved challenges in unified representations, multimodal queries, and modality-specific adaptation.

Abstract

from arXiv · show

The rapid growth of large language models (LLMs) with diverse capabilities, costs, and domains has created a critical need for intelligent model selection at inference time. While smaller models suffice for routine queries, complex tasks demand more capable models. However, static model deployment does not account for the complexity and domain of incoming queries, leading to suboptimal performance and increased costs. Dynamic routing systems that adaptively select models based on query characteristics have emerged as a solution to this challenge. We provide a systematic analysis of state-of-the-art multi-LLM routing and cascading approaches. In contrast to mixture-of-experts architectures, which route within a single model, we study routing across multiple independently trained LLMs. We cover diverse routing paradigms, including query difficulty, human preferences, clustering, uncertainty quantification, reinforcement learning, multimodality, and cascading. For each paradigm, we analyze representative methods and examine key trade-offs. Beyond taxonomy, we introduce a conceptual framework that characterizes routing systems along three dimensions: when decisions are made, what information is used, and how they are computed. This perspective highlights that practical systems are often compositional, integrating multiple paradigms under operational constraints. Our analysis demonstrates that effective multi-LLM routing requires balancing competing objectives. Choosing the optimal routing strategy depends on deployment and computational constraints. Well-designed routing systems can outperform even the most powerful individual models by strategically leveraging specialized capabilities across models while maximizing efficiency gains. Meanwhile, open challenges remain in developing routing mechanisms that generalize across diverse architectures, modalities, and applications.

1 Introduction

Dynamic multi-LLM routing addresses the cost-performance dilemma caused by heterogeneous query difficulty by selecting or escalating among independently trained models. The survey organizes methods by routing paradigms and by when decisions occur, what signals they use, and how decisions are computed.

  • Queries vary from simple factual requests to complex multi-step reasoning, making uniform deployment inefficient or inadequate.
  • Dynamic routing analyzes each query and selects among models with different capabilities, costs, and specializations.The goal is to match computational resources to query requirements while reducing costs and maintaining or improving quality.
  • Routing selects one model before generation, whereas cascading sequentially escalates from smaller models when response quality is insufficient.Production systems often combine both approaches.
  • The survey covers six paradigms: difficulty-aware, preference-aligned, clustering-based, reinforcement learning, uncertainty-based, and cascading methods.It also discusses multimodal routing, evaluation frameworks, and future directions.
  • The framework characterizes systems by decision timing, information signals, and computational mechanism.These dimensions include pre-generation, post-generation, or multi-stage decisions; query, model, response, or feedback signals; and heuristic, supervised, bandit, or reinforcement-learning computation.

2 Difficulty-aware Routing

Difficulty-aware routing estimates query complexity or query-model compatibility before selecting a model, typically balancing accuracy, cost, latency, or reasoning effort. Methods range from heuristics and classifiers to learned capability representations and graph-based prediction.

  • Difficulty-aware routing directs simpler queries to smaller models and complex reasoning tasks to more capable models.Difficulty can be estimated from text length, rarity, idiomatic language, syntactic complexity, classifiers, or LLM judgments.
  • The design-space matrix distinguishes routing systems by timing, signals, and computation, providing context for difficulty-aware methods.Its axes include pre-generation or post-generation decisions, query or response signals, and heuristic, supervised, bandit, or RL computation.
  • BEST-Route uses a DeBERTa-v3-small multi-head router to estimate difficulty and select both the model and sampling strategy.It applies best-of-n sampling at inference time.
  • vLLM Semantic Router classifies query intent and complexity, enabling reasoning-capable inference only for queries that require it.Straightforward queries use standard inference to preserve low latency and efficiency.
  • RouteLMT uses lightweight LoRA adaptation to estimate the large model’s marginal gain over a small translator for routing.The router probes prompt-token representations without external models or hypothesis decoding.
  • EmbedLLM, ICL-Router, GraphRouter, and IRT-Router predict query-model compatibility using model representations, historical performance, costs, or query properties.These approaches avoid relying solely on explicit difficulty labels, although some require retraining when models change.

3 Human Preference-aligned Routing

Preference-aligned routing uses human or synthetic preference information to choose models according to quality-cost trade-offs or explicit user policies. Arch-Router extends this idea by conditioning routing on user-defined domain-action preferences supplied at inference time.

  • RouteLLM trains a router to choose between strong and weak LLMs using preference data while optimizing response quality against cost.Its routing decision is formulated as a binary choice between high-quality, high-cost and lower-quality, low-cost models.
  • Arch-Router supplies routing policies and the user conversation to a router that selects the appropriate policy and corresponding LLM.The figure illustrates this mechanism for coding usage.
  • Arch-Router lets users define domain-action pairs so different request types can be assigned to different models.Routing policies are provided in the input context to a 1.5B-parameter model, allowing preference updates without retraining.
  • Prompt-to-Leaderboard generates prompt-specific Bradley-Terry coefficients to rank models for each user query rather than averaging performance across tasks.The resulting scores support task-specific evaluation and cost-constrained selection.
  • Eagle combines global and local ELO rankings to estimate overall and specialized model capabilities without training a router.The local component evaluates performance on similar past queries.
  • Zooter trains a routing classifier with labels derived from the QwenRM reward model’s scores of candidate responses.

4 Clustering-based Routing

Clustering-based routing groups similar queries and assigns each cluster to a cost-effective LLM without requiring explicit task labels. UniRoute profiles candidate models by cluster performance, while Avengers-Pro uses cluster allocation to improve the accuracy-cost frontier.

  • Clustering-based routing groups similar queries and assigns each group to its most suitable LLM using unsupervised learning.The approach balances performance and cost without requiring explicit task labels.
  • UniRoute applies K-means to identify query centroids, evaluates each LLM’s cluster-level performance, and routes using cost-adjusted error.Incoming prompts are assigned to clusters, with dynamic cost adjustment controlling the accuracy-cost trade-off.
  • UniRoute can add new LLMs at inference time without retraining by evaluating them on existing clusters.It operates from query embeddings and cluster assignments and can route across unseen LLMs.
  • Avengers-Pro establishes a Pareto frontier that surpasses GPT-5-medium with better accuracy at lower cost through strategic cluster-based allocation.

5 Reinforcement Learning Routing

Reinforcement-learning routers frame model selection as sequential or online decision-making, using policy optimization and bandit feedback to balance quality, cost, latency, and energy. These methods differ in whether they decompose tasks, learn performance estimators, incorporate feedback, or adapt to changing model behavior.

  • Overview: RL routing treats model selection as decision-making under uncertainty, combining multi-step policy optimization with online exploration and exploitation.Policy methods refine routing through multi-step interactions, while bandit methods adapt from deployment feedback.
  • Policy Optimization Methods: Router-R1 alternates internal reasoning with model assignment for up to four routing steps, using PPO-trained routing over Qwen2.5-3B-Instruct and LLaMA-3.2-3B-Instruct.Its router can select and aggregate responses and is conditioned on pricing and latency descriptors.
  • Policy Optimization Methods: R2-Reasoner decomposes complex tasks into subtasks, allocates each to a suitable model, and reports 84.46% API cost savings while maintaining competitive reasoning accuracy.Its staged supervised-fine-tuning and GRPO pipeline avoids end-to-end gradient propagation and generalizes to unseen models.
  • Trade-offs: Multi-step RL routing can incur inference latency because reasoning and decomposition require multiple model calls, favoring settings where cost savings outweigh latency concerns.The passage specifically highlights commercial APIs with high per-call costs as a suitable setting.
  • Bandit-Based Methods: Bandit routers adapt model assignments through feedback, with MixLLM reaching 97.25% of GPT-4’s quality at 24.18% of the cost under time constraints.MixLLM uses domain-aware query representations, binary user feedback, and continual policy-gradient updates.
  • Bandit-Based Methods: Other bandit designs target preference comparison, energy efficiency, or nonstationary rewards through pairwise feedback, direct GPU measurements, or convergence detection.GreenServ reports a 22% accuracy gain and 31% lower cumulative energy than random routing, with under 8 ms routing overhead per query.

6 Uncertainty-based Routing

Uncertainty-based routing estimates whether model confidence aligns with correctness so systems can escalate uncertain queries. Evidence favors probe- and perplexity-based estimates over verbalized confidence, while self-verification can help when embedded in cascades.

  • Motivation: Uncertainty quantification supports routing by identifying when an initial model should escalate a query to a more capable model.The central requirement is alignment between estimated confidence and actual correctness.
  • Probing and Probability-based Uncertainty Estimation: Probe-based classifiers and perplexity-based methods significantly outperform verbalization approaches for routing from SLMs to LLMs on edge devices.The benchmark evaluated eight uncertainty quantification methods.
  • Probing and Probability-based Uncertainty Estimation: SLMs match LLM performance on high-confidence queries comprising the top 20%, while trained hidden-state probes provide reliable confidence estimates at the cost of weight access and supervision data.These findings are reported alongside prior experiments by Mahaut et al.
  • Conformal Prediction: CP-Router applies conformal prediction to route between standard LLMs and verbose Large Reasoning Models using answer-option probabilities and calibrated uncertainty scores.For multiple-choice questions, uncertainty is computed as one minus each option’s probability.
  • LLM-as-a-Judge Uncertainty Estimation: One-step and two-step verbalization show consistently low alignment between reported uncertainty and prediction correctness.The two variants ask for confidence alongside the answer or in a follow-up query.
  • Cascading: Self-verification can be effective in cascading systems when combined with other mechanisms, as illustrated by AutoMix and Self-REF.AutoMix uses few-shot in-context learning, whereas Self-REF uses lightweight fine-tuning for confidence calibration.

7 Cascades

Cascading begins with smaller, cheaper models and escalates, refines, or rejects responses when quality estimates indicate that stronger models are needed. Surveyed systems use external quality estimation, self-verification, and dynamic model ordering to balance quality and efficiency.

  • Cascading foundations: Cascading queries smaller models first and escalates to larger models when the initial response is insufficient.This adapts resource allocation to query difficulty while involving multiple models in one answer process.
  • Self-verification: AutoMix generates with a small model, self-verifies its answer, and routes to a larger model using the resulting confidence assessment.The procedure repeats across models until a final answer is reported and targets a user-specified cost-quality tradeoff.
  • Cascade routing: Cascade routing dynamically skips or reorders models at each step instead of following a fixed smallest-to-largest sequence.The survey identifies quality estimation as critical and reports that cascade routing consistently outperforms individual approaches.
  • Self-verification: Self-REF fine-tunes confidence tokens for downstream routing, sending low-confidence queries to stronger models or rejecting them in favor of safe default behavior.This differs from AutoMix, which relies on few-shot in-context learning without fine-tuning.
  • Ensemble approaches: LM-Blender queries multiple models simultaneously, ranks candidate responses with a Pair Ranker, and merges top outputs through a Gen Fuser.It represents ensemble-based cascading alternatives that combine responses rather than only escalating sequentially.
  • Quality estimation: Quality estimation can defer only translation segments failing quality criteria to larger models, reducing large-model calls while maintaining quality.Quality-estimation metrics can select models globally or for individual source-text segments.

8 Multimodal Model Routing

Multimodal routing extends model selection beyond text, but modality-specific representations create additional challenges. Existing work includes model-pool ranking, multimodal probing, and benchmarks for evaluating routing under fixed candidates and costs.

  • Motivation: Text-focused routing approaches may extend to speech, image, and video, but differing modality representations pose new challenges.The survey identifies multimodal representation as a central obstacle to directly transferring text-based routing methods.
  • Model selection: Model-Spider ranks visual models and LLMs by predicted downstream-task fitness rather than dynamically dispatching queries at inference time.It evaluates candidate models from a large pool and selects according to predicted task suitability.
  • Multimodal routing methods: ReLope addresses degraded hidden-state probes in multimodal LLMs with complementary techniques designed for visual-input routing.Visual tokens weaken the separability of correctness signals that probes exploit in text-only models.
  • Evaluation: MMR-Bench evaluates multimodal routing across OCR, general VQA, and multimodal math reasoning using strong baselines, oracle bounds, and representative policies.Its comparisons use fixed candidate sets and cost models.
  • Open challenges: Multimodal routing remains underexplored, with open challenges in unified representations, multi-modality queries, and modality-specific routing strategies.The survey highlights extending difficulty estimation, uncertainty quantification, and preference alignment beyond text-only settings.

9 Evaluation

Routing evaluation combines benchmarks with metrics for response quality, decision quality, efficiency, cost, and environmental impact. Large benchmark resources support comparison across models, tasks, difficulty levels, and operating constraints.

  • Routing benchmarks: RouterBench provides over 405k precomputed outputs from eleven LLMs across seven tasks with performance and cost metadata.The dataset enables routing analysis under realistic constraints.
  • Routing benchmarks: RouterEval contains over 200 million performance records from more than 8,500 LLMs across 12 benchmarks and evaluates routing across model-group compositions and difficulty levels.It frames routing as m-way classification for m = 3, 5, 10, 100, and 1000.
  • Routing benchmarks: MixInstruct aggregates 110k instruction examples and supplies oracle pairwise preferences for evaluating routing and ensemble methods.Its supervision is based on LLM comparisons across multiple instruction datasets.
  • Routing benchmarks: LLMRouterBench contains over 400K instances from 21 datasets and 33 models, with metrics for performance routing and performance-cost trade-offs.It integrates 10 routing baselines and standardized collector, evaluator, and adaptor components.
  • Efficiency and cost metrics: Evaluation also considers routing accuracy, preference win rate, AUC across operating points, latency, throughput, cost, quality-cost Pareto frontiers, energy, and carbon footprint.Routing overhead must remain negligible while systems balance computational efficiency and response quality.
  • Performance and quality metrics: Task performance measures whether routed responses are correct or high quality, using metrics such as accuracy, exact match, pass@k, chrF, COMET, and LLM-as-a-Judge scores.Overall routing performance aggregates task performance across all routed queries.

10 Towards Multidimensional Routing Systems

The survey characterizes practical routing as multidimensional and compositional, combining decisions, signals, and computation across a control pipeline. It also identifies gaps in online response-level adaptation, learned escalation, and unified multi-objective optimization.

  • Compositional systems: Production routing systems compose mechanisms across paradigms to address heterogeneous quality, latency, cost, and safety constraints.They combine pre-generation and post-generation decisions, query- and response-level signals, and offline learning with online adaptation.
  • Control pipeline: A representative control pipeline uses a low-cost pre-router, a post-generation verifier, and an escalation policy that accepts, refines, rejects, or defers responses.This pipeline treats cascades as a natural extension of routing systems.
  • Control pipeline: FrugalGPT and Cascade Routing instantiate the three pipeline stages by selecting an initial model, evaluating its response, and deciding whether to accept or defer.Their compositional pattern combines query and cost signals with response quality estimation.
  • Method characterization: Routing methods should report when decisions occur, what signals they use, and how decisions are computed.The framework distinguishes pre-generation, post-generation, or multi-stage timing; query, metadata, response, or feedback signals; and heuristic, supervised, bandit, or RL computation.
  • Open gaps: No current method simultaneously combines response-level signals with online adaptation, while learned escalation policies and unified multi-objective optimization remain limited.These gaps concern combining deployment feedback with response signals and jointly optimizing quality, cost, and latency.
  • Operational evaluation: Efficient inference frameworks can report TTFT, TPOT, throughput, and goodput under constraints such as concurrency and maximum token budget.These measurements support operational evaluation of multidimensional routing systems.

11 Conclusion and Future Directions

The survey synthesizes dynamic multi-LLM routing research and identifies generalization, multi-stage cascading, and multimodality as open directions. It argues that routing can combine model complementarity and specialization with efficiency gains, while future systems should reflect increasingly diverse deployment settings.

  • The survey organizes dynamic multi-LLM deployment methods into six paradigms and examines multimodal routing, evaluation frameworks, benchmarks, and metrics.
  • Generalization: Routing methods often struggle to generalize beyond fixed LLM sets, creating a need for retraining-free transfer across architectures, tasks, and deployment scenarios.
  • Multi-stage cascades: Future work should study multi-stage cascades that process queries and outputs at multiple levels while balancing quality, efficiency, and safety.
  • Multimodality: Multimodal routing remains underexplored, especially for unified representations, multi-modality queries, modality-specific characteristics, and cross-modal computational costs.
Loading 2603.04445v2…