PaperlayerView exact revision on arXiv

Source-linked AI summary

RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning

Shihong Huang, Shengjie Wang, Hong Ma, Zhou Xu

arXiv:2608.15817v1cs.AI

TL;DR

Efficiently routing queries across LLMs is difficult because models differ in capability and inference cost, while existing routers either commit before observing responses or follow fixed model orders. RLCascadeRouter directly learns stop-or-select decisions with reinforcement learning and achieves strong performance-cost trade-offs across ten benchmarks, including generalization to unseen models without retraining.

  • Problem

    Heterogeneous LLM capabilities and inference costs make selecting a satisfactory model or model sequence while controlling inference cost challenging.

  • Method

    RLCascadeRouter formulates cascade routing as a decision-aligned MDP and jointly learns stopping and model selection from trajectory-level performance-cost feedback.

  • Results

    Across ten LLMRouterBench benchmarks, RLCascadeRouter achieves strong performance-cost trade-offs, generalizes to unseen models without policy retraining, and its policy components are effective.

  • Takeaways & Limitations

    Cascade routing can be optimized directly for the final performance-cost objective without independently trained response-quality estimators.

  • Takeaways & Limitations

    The routing objective uses candidate-model minimum and maximum scores and a maximum cost defined by the allowed routing depth.

Abstract

from arXiv · show

The growing ecosystem of large language models (LLMs) offers huge potential to optimize performance-cost trade-offs. However, their heterogeneous capabilities and inference costs make efficiently routing queries a significant challenge. Existing paradigms are inflexible: one-shot routers commit before observing responses, whereas conventional cascades stop adaptively but follow a fixed model order. Cascade routing removes both restrictions by reconsidering whether to stop or invoke another model after each response. Current methods use a predict-then-optimize pipeline estimating response quality and future model utility. However, prediction loss for quality or utility is not equivalent to routing-decision loss. A lower prediction error does not necessarily yield a better action; a small boundary-crossing error can reverse a ``stop'' or model-selection decision. Therefore, we propose RLCascadeRouter, a quality-estimator-free framework that formulates cascade routing as a Markov decision process with actions comprising ``stop'' and model selection. It uses trajectory returns and advantages to directly optimize the performance-cost objective. Its Cascade Policy Network models candidate complementarity for model selection and remaining-action value for stopping, eliminating independent post-hoc response-quality estimators. Evaluated across ten LLMRouterBench benchmarks with thirteen LLMs, RLCascadeRouter outperforms strong baselines and achieves superior performance-cost trade-offs. It incorporates unseen models without retraining, and ablation studies validate both policy components.

1 Introduction

LLM heterogeneity creates a need to balance answer quality against inference cost, but existing routing and cascading methods impose rigid decisions or depend on quality estimation. RLCascadeRouter instead learns stopping and model-selection actions directly through reinforcement learning in a unified MDP.

  • Motivation: Heterogeneous LLM capabilities and inference costs make selecting a satisfactory model or model sequence while controlling inference cost necessary.Large general-purpose models can be expensive for every query, while smaller or specialized models vary across tasks and instances.
  • Limitations of Existing Methods: One-shot routing commits to a candidate model, whereas fixed-order cascading invokes models sequentially until a response is considered sufficient.These are the two main existing paradigms for LLM model selection.
  • Limitations of Existing Methods: Cascade routing generalizes routing and cascading by dynamically choosing between stopping and future model combinations, but remains dependent on ex-ante and post-hoc quality estimation.This predict-then-optimize design separates quality prediction from the routing decision.
  • Proposed Framework: RLCascadeRouter formulates cascade routing as an MDP whose actions jointly include stopping and all previously unselected models, optimizing action preferences from trajectory-level performance-cost feedback.The framework avoids predicting response qualities and converting those predictions into routing decisions.
  • Policy and Evaluation: Its Cascade Policy Network models remaining-model complementarity and value-aware stopping to learn when to stop and which model to invoke next.Experiments on LLMRouterBench report strong performance-cost trade-offs, generalization to unseen models without policy retraining, and effective policy components.

2 Related Work

Prior routing methods trade off pre-generation model choice against post-generation cascading, but each limits adaptivity in one dimension. RLCascadeRouter unifies both by treating stopping and selecting any unselected model as peer actions after each response.

  • Pre-generation routing: Early routers select a model before generation, typically using query difficulty or related signals to assign inputs across models with different capabilities and costs.Examples include HybridLLM and other model-selection methods for natural-language queries with known answers.
  • Conventional cascading: Cascading methods invoke models successively and use response reliability signals to decide whether to return the current answer or continue.FrugalGPT jointly learns an ordered cascade and response scorers, while related approaches use confidence, consistency, or quality estimates.
  • Conventional cascading: Conventional cascades vary call count but usually preserve a predetermined escalation order and delegate stopping to a confidence score, consistency rule, or quality estimator.This limits which model can be chosen next even when post-generation feedback is available.
  • Adaptivity gap: Single-step routers choose broadly but cannot revise after generation, whereas conventional cascades can reconsider answers but cannot freely choose the next model.The two paradigms therefore provide complementary but incomplete forms of adaptivity.
  • RLCascadeRouter: RLCascadeRouter treats “stop” and every unselected candidate model as peer actions at each decision step, allowing cascade order and depth to depend on the query and observed response.This combines adaptive stopping with free next-model selection rather than fixing either decision in advance.

3 RLCascadeRouter

RLCascadeRouter avoids the mismatch between response-quality prediction and routing decisions by directly learning cost-aware stopping and model-selection actions in an MDP. Its Cascade Policy Network jointly models candidate complementarity and stopping value without requiring a quality estimator or inference-time stopping threshold.

  • 3.1 Prediction versus decision: Prediction loss and routing-decision loss are not equivalent: a lower-loss utility estimate can reverse the optimal action, while a higher-loss estimate preserves it.With δ = 0.10 and K = 0.15, the lower-loss prediction reverses CONTINUE/STOP ordering, whereas the higher-loss prediction preserves the correct decision.
  • 3.2 MDP formulation: RLCascadeRouter formulates model selection and stopping over candidate LLMs as an MDP with cost-aware rewards, adaptive routing depth, and actions for STOP or unselected models.STOP is masked initially because no response exists, and previously selected models are masked to prevent repeated invocation.
  • 3.2 MDP formulation: The policy maximizes cumulative performance-cost utility, with incremental rewards designed so that γ = 1 makes trajectory return equal final normalized utility.Performance and cumulative API cost are normalized using query-specific score extrema and the maximum allowed routing cost.
  • 3.2 MDP formulation: The MDP jointly learns model order and cascade depth per query, deciding whether another model’s expected benefit justifies its cost without a quality estimator or stopping threshold.At each step, the policy chooses between returning the current response and invoking any previously unselected model.
  • 3.3 Cascade Policy Network: The Cascade Policy Network uses a Complementarity Encoder to identify valuable continuations and a Value-Aware Stopper to compare stopping with model selection.The two modules are jointly trained and correspond to the Bellman comparisons for continuation value and current terminal value.

4 Experimental Setup

Experiments use ten LLMRouterBench benchmarks spanning mathematics, code, software engineering, knowledge, instruction following, and tool use. Methods are evaluated by macro-averaged performance and cumulative cascade API cost against individual-model and routing baselines.

  • Datasets and Metrics: The ten benchmarks cover mathematical reasoning, code generation, software engineering, knowledge and general reasoning, instruction following, and tool use.They include AIME and LMB; LCB and SWE; GPQA, HLE, MMLU-Pro, and SQA; ArenaHard; and Tau2.
  • Datasets and Metrics: Performance is the macro-average of mean response scores across the ten datasets, while cost accumulates API costs for every model call in the complete cascade trajectory.Performance scores use the response from the last selected model and include intermediate calls in cost accounting.
  • Baselines: RLCascadeRouter is compared with 13 individual LLMs and five representative routing strategies, including Random Router, HybridLLM, FrugalGPT, and GraphRouter.The individual-model baselines include GPT-5, GPT-5-Chat, Gemini-2.5-Pro, Gemini-2.5-Flash, Claude-Sonnet-4, Qwen3 variants, DeepSeek variants, GLM-4.6, Kimi-K2-0905, and Intern-S1.
  • Implementation Details: Queries are split 60%/10%/30% into training, validation, and testing with seed 42, and each query permits at most three model calls.Optimization uses AdamW with learning rate 3 × 10^-5 and weight decay 10^-4, alongside the specified PPO clipping, discount, value-loss, entropy, and gradient-clipping settings.

5 Experimental Analysis

RLCascadeRouter achieves the strongest aggregate performance and a more favorable performance-cost frontier than routing and individual-model baselines. Its unseen-model evaluation and ablations support generalization and the value of modeling candidate complementarity and remaining-action context.

  • Overall performance: 68.81% macro-average performance makes RLCascadeRouter the best method, exceeding Avengers-Pro by 0.97 points and GPT-5 by 3.69 points at α = 1.0.Table 1 compares five routing methods and all 13 individual LLMs.
  • Performance-cost trade-off: At near-59% performance, RLCascadeRouter reaches 59.11% at cost 10.64, versus Avengers-Pro’s 59.10% at 16.12.Across most of the medium- and high-performance range, RLCascadeRouter provides a more favorable non-dominated frontier.
  • Unseen-model generalization: Without retraining, the original policy retains 90.99% of initial-pool performance after replacing three frequently selected models, achieving 53.97% average performance.The initial pool achieves 59.32% performance; all settings use α = 1.0.
  • Unseen-model generalization: Replacement models are selected for 98.18% of queries, showing that textual model descriptions enable the router to incorporate unseen candidates without policy retraining.Setting B evaluates the original-policy checkpoint on the modified pool, while Setting C retrains and evaluates on that pool.
  • Ablation analysis: Full exceeds w/o CE by 0.51, 1.59, and 1.21 points at α = 1.0, 0.5, and 0.2, respectively, supporting candidate-complementarity modeling.Removing VAS reduces performance by 1.38 and 1.23 points at α = 1.0 and 0.5, but improves it by 0.60 points at α = 0.2; QE Stop reduces performance by 1.11 and 0.67 points.

6 Conclusion

RLCascadeRouter replaces quality estimation with a decision-aligned MDP policy whose Cascade Policy Network jointly supports model selection and stopping. Across ten benchmarks, it delivers strong performance-cost trade-offs, generalizes to unseen models without retraining, and validates its policy design.

  • 6 Conclusion: RLCascadeRouter replaces predict-then-optimize quality estimation with a decision-aligned MDP policy for cascade routing.Its Cascade Policy Network jointly learns candidate complementarity for model selection and remaining-action value for stopping.
  • 6 Conclusion: Ten benchmarks demonstrate strong performance-cost trade-offs, generalization to unseen models without retraining, and the effectiveness of the proposed policy design.

A Cascade Policy Network Details

The Cascade Policy Network (CPN) parameterizes both the routing policy and value function through two components: a Complementarity Encoder for comparing remaining models and a Value-Aware Stopper for deciding whether continuation is worthwhile.

  • Network role: The CPN parameterizes the stochastic policy πθ(a_t | s_t) and value function Vϕ(s_t) of the routing MDP.Its inputs and outputs support both action selection and state-value estimation.
  • Complementarity Encoder: The Complementarity Encoder compares the remaining model actions to inform model selection.It represents candidate complementarity among models still available in the cascade.
  • Value-Aware Stopper: The Value-Aware Stopper determines whether the best continuation opportunity justifies another model call.This component supports adaptive stopping by weighing the value of continuing against stopping.

A.1 State Representation … A.4 Complementarity Encoder

RLCascadeRouter represents each cascade decision with query, response, model-history, cost, depth, and availability information, then augments it with query-conditioned priors and masked action tokens. A complementarity encoder contextualizes valid actions jointly so shared actors score model continuation choices by relative contribution rather than isolated compatibility.

  • A.1 State Representation: The state combines query and current-response embeddings with selected-model history, most-recent model, dataset, cost, depth, response availability, and selection-fraction features.This distinguishes unanswered queries from queries with strong responses and records accumulated routing context; initial response and model-history embeddings are zero vectors.
  • A.2 Query-Model Prior: A training-split-only Query-Model Prior combines soft query-type membership with smoothed model performance-cost statistics.The statistics include performance, accuracy, cost, score percentile, and observation count, and the prior enters as an input feature rather than an external routing rule.
  • A.3 Action Representation: The action universe contains STOP and model-selection actions, with validity masking enforcing when each action can be chosen.STOP requires a prior response; previously selected models and model calls at maximum depth are unavailable.
  • A.3 Action Representation: Each valid model action includes response and history similarities, query-model prior information, and normalized expected invocation cost.Similarities involving unavailable response or history features are set to zero.
  • A.3 Action Representation: STOP uses the shared state representation without model-specific features, while tokenized actions allow shared parameters across candidate models.The STOP token is xt,stop = [ft; 0; 0; 1].
  • A.4 Complementarity Encoder: The Complementarity Encoder contextualizes every valid action against the complete valid-action set using shared hidden projections, state fusion, and Transformer self-attention.This design reflects that a model’s marginal value depends on the current response, selected models, and remaining alternatives.
  • A.4 Complementarity Encoder: Self-attention makes each model representation encode relative contribution, and a shared ModelActor scores continuation actions without fixed model-index output heads.The continuation score is ρt,i = ModelActor(ht,i), mi ∈At.

A.5 Value-Aware Stopper

The Value-Aware Stopper (VAS) summarizes remaining action opportunities through masked mean pooling and combines them with the current state. Its stop preference is learned jointly with model selection from trajectory-level performance-cost returns, without predicting absolute quality or using calibrated thresholds.

  • VAS summarizes remaining action opportunities by masked mean pooling.
  • The pooled candidate-set representation is concatenated with the current state before computing the Stop Gate and value critic.
  • The critic estimates policy advantages during training but does not directly determine the selected action.
  • VAS avoids absolute-quality prediction and calibrated thresholds by learning stopping preference jointly with model selection from trajectory-level performance-cost returns.

A.6 Joint Routing Policy

RLCascadeRouter uses a single hierarchical policy that combines the Candidate Evaluator and Value Assessment Subnetwork into masked routing decisions. At inference, it either returns the current response or invokes a selected model without ground-truth scores or an independent post-hoc quality estimator.

  • Joint Routing Policy: The Candidate Evaluator and Value Assessment Subnetwork form a single hierarchical policy with unified action logits.Their outputs are combined before applying the validity mask.
  • Joint Routing Policy: The Value Assessment Subnetwork controls the global stop-versus-continue preference, while the Candidate Evaluator selects the model conditional on continuation.The resulting action distribution is validity-masked before routing.
  • Joint Routing Policy: At inference, the masked policy either returns the current response or invokes the selected model without ground-truth response scores or an independent post-hoc quality estimator.This directly supports routing without separate response-quality estimation.

B Model Descriptions

RLCascadeRouter represents each candidate model with an embedding of its textual description. The section provides exact descriptions for all 13 candidate models used in the experiments.

  • Model representation: Each candidate model is represented by an embedding of its textual description.This representation is used for the 13 candidate models in the experiments.
Loading 2608.15817v1…