PaperlayerView exact revision on arXiv

Source-linked AI summary

R2-Router: A New Paradigm for LLM Routing with Reasoning

Jiaqi Xue, Qian Lou, Jiarong Xing, Heng Huang

arXiv:2602.02823v2cs.CL

TL;DR

Existing routers treat each LLM as having one fixed quality–cost profile, overlooking quality changes with output length. R2-Router jointly reasons over the LLM and output budget, supported by R2-Bench’s multi-budget data, and achieves comparable quality at 4 −5× lower cost than prior methods.

  • Problem

    Existing routers assume fixed quality and cost per LLM for each query, overlooking how output length changes the same LLM’s quality and cost.

  • Method

    R2-Router predicts each LLM’s quality across discrete cost budgets and jointly selects the model and a length-constrained output budget; R2-Bench records these behaviors across diverse lengths.

  • Results

    4 −5× lower cost: R2-Router achieves comparable quality to existing routers and reaches higher quality at a given cost across evaluations.

  • Takeaways & Limitations

    Routing can search over LLM–budget combinations rather than fixed model choices, exposing efficient operating points that prior routers overlook.

  • Takeaways & Limitations

    “Reasoning” denotes the router’s decision process over LLM–budget pairs, not LLM-internal chain-of-thought reasoning.

Abstract

from arXiv · show

As LLMs proliferate with diverse capabilities and costs, LLM routing has emerged by learning to predict each LLM's quality and cost for a given query, then selecting the one with high quality and low cost. However, existing routers implicitly assume a single fixed quality and cost per LLM for each query, ignoring that the same LLM's quality varies with its output length. This causes routers to exclude powerful LLMs when their estimated cost exceeds the budget, missing the opportunity that these LLMs could still deliver high quality at reduced cost with shorter outputs. To address this, we introduce R2-Router, which treats output length budget as a controllable variable and jointly selects the best LLM and length budget, enforcing the budget via length-constrained instructions. This enables R2-Router to discover that a powerful LLM with constrained output can outperform a weaker LLM at comparable cost-efficient configurations invisible to prior methods. Together with the router framework, we construct R2-Bench, the first routing dataset capturing LLM behavior across diverse output length budgets. Experiments show that R2-Router achieves state-of-the-art performance at 4-5\times lower cost compared with existing routers. This work opens a new direction: routing as reasoning, where routers evolve from reactive selectors to deliberate reasoners that explore which LLM to use and at what cost budget. The code is publicly available at https://github.com/UCF-ML-Research/R2-Router.

1. Introduction

LLM routing must balance varying model capabilities and costs, but existing routers overlook how output length changes quality and cost. R2-Router jointly selects an LLM and length budget, while R2-Bench captures this behavior for training and evaluation.

  • LLM routing addresses the challenge of selecting a model that balances response quality and inference cost for each query.
  • Existing routers assume each LLM has a fixed quality–cost profile, potentially excluding powerful models whose unconstrained cost exceeds the budget.The same model can still provide satisfactory quality at lower cost when its output length is constrained.
  • R2-Router predicts quality across output lengths, then jointly selects an LLM and a budget enforced through length-constrained instructions.This changes routing from choosing among fixed model profiles to searching over model–budget combinations.
  • R2-Bench records each LLM’s behavior across diverse output lengths for the same queries, enabling routers to model quality as a function of cost.
  • 4 −5× lower cost: R2-Router reaches comparable quality to existing methods, while an integrated UniRouter case improves AUDC by 5% and reduces cost by 80%.The reported experiments also include plug-in integration with existing routers and performance on a public routing leaderboard.

2. Related Work

Prior work spans cost-ordered cascades, predictive routing, inference-mode selection, and routing benchmarks. However, existing benchmarks generally record one response per LLM and therefore omit behavior across output lengths, which R2-Bench targets.

  • Inference-time scaling studies report that longer outputs can improve quality but increase cost, with gains often saturating beyond a threshold.Concise instructional prompts are discussed as a way to reduce output length while maintaining comparable quality.
  • Open-source and commercial platforms increasingly deploy routers to select models for specific inputs and balance capability against pricing tiers.
  • Routing systems include cost-ordered cascades, predictive model selectors, and approaches that route inference modes or reasoning strategies.Cascades may incur multiple model calls and added latency, while other methods extend routing beyond selecting a single model.
  • Existing routing benchmarks aggregate large numbers of inference outcomes, but they generally record only one response per LLM for each query.
  • R2-Bench fills this gap by capturing the same LLM’s behavior across diverse output lengths, enabling routing based on quality variation with cost.

3. Dataset Construction: R2-BENCH

R2-Bench collects multiple budgeted responses per model and query to represent quality–cost curves rather than single points. This supports finer-grained Oracle selection over both models and output lengths, producing higher measured upper bounds than prior data.

  • R2-Bench records multiple responses under different output-length budgets for each LLM and query, using standard training and evaluation splits.
  • The dataset combines queries from six benchmarks spanning reasoning, knowledge understanding, mathematics, and retrieval-augmented generation, with a fixed pool of 15 open-source LLMs.
  • Qwen3-80B-Instruct was selected as the R2-Bench judge after achieving the highest correlation with human judgments, ρ = 0.82, among four candidates.
  • R2-Bench represents each LLM as a quality–cost curve, allowing the Oracle to select both the model and its output length under a trade-off coefficient λ.
  • AUDC improves from 0.85 to 0.98, QNC decreases from 0.18 to 0.04, and Peak Quality increases from 0.90 to 0.98 versus SPROUT2.The gain comes from selecting optimal costs per LLM in addition to selecting among LLMs.

4. Routing as Reasoning: R2-ROUTER

R2-ROUTER extends routing from choosing among fixed LLM quality–cost points to jointly selecting an LLM and controllable output budget. It predicts quality–cost curves and searches this expanded space to balance quality and cost.

  • From Reaction to Reasoning: Existing routers assign each query to an LLM using fixed quality and cost estimates, restricting their search to one operating point per model.
  • From Reaction to Reasoning: R2-ROUTER treats output length as a controllable variable and selects the optimal (LLM, token budget) pair under a cost-sensitive objective.Budget constraints are enforced through length-constrained instructions.
  • From Reaction to Reasoning: The expanded search can retain powerful models that fixed-cost routing would reject, allowing a constrained-budget large model to outperform a smaller model while satisfying the cost constraint.
  • Architecture and Routing: R2-ROUTER uses a shared encoder with independent cost-specific quality-prediction heads, while interpolation can extend sparse anchor costs to a continuous budget range.The predictors estimate each LLM’s quality at different costs, and the resulting curves support routing decisions.
  • R2-BENCH: R2-BENCH records multiple responses for each query–LLM pair under diverse output budgets, enabling quality–cost curves for training and evaluation.
  • Extensions: The curve-based approach also integrates with UniRouter, improving its performance while preserving support for dynamically changing LLM pools.

5. Experimental Setup

The experiments use R2-BENCH, a diverse benchmark with multiple LLMs and controlled output budgets, and compare R2-ROUTER against reactive routing baselines. Evaluation traces quality–cost trade-offs using deferral curves and multiple metrics across repeated runs.

  • Dataset: R2-BENCH contains 30,968 queries across 20 categories from 6 benchmarks, covering mathematical reasoning, general knowledge, graduate-level science, and other domains.
  • Dataset: Each query is paired with responses from 15 LLMs under 16 token-budget levels, with quality judged on a 0-to-1 scale and costs controlled through prompts and truncation.
  • Models: The heterogeneous LLM pool spans 0.6B to 235B parameters and includes both general-purpose and domain-specific models.
  • Baselines: R2-ROUTER is compared with leading RouterArena methods, whose baselines predict fixed quality and cost without modeling cost-dependent quality.
  • Implementation: Training uses 1024-dimensional query embeddings and separate three-layer MLP predictors for each cost level, optimized with MSE and Adam for 100 epochs.
  • Implementation: Router training is lightweight, taking approximately 30 minutes for 15 LLMs on a single NVIDIA RTX 3090.
  • Evaluation: Evaluation uses deferral curves plotting average response quality against total inference cost, with results reported over 5 independent random-seed runs.The reported cost includes both input and output tokens.

6. Results

R2-ROUTER consistently improves the quality–cost trade-off, generalizes to new and out-of-distribution models and queries, and remains effective across design choices. Its curve-based routing also retains strong performance with few interpolation heads and under alternative evaluation settings.

  • Main Results: R2-ROUTER reaches average quality 0.8 at approximately 0.5×10^-3 cost, while reactive baselines require 4× to 5× more budget for similar performance.It leads all baselines across seven RouterArena categories under exact-match scoring.
  • Generalization to New LLMs: UNI-R2ROUTER achieves higher AUDC than UniRouter on an expanded pool of unseen models, 0.623 versus 0.590, while reducing QNC by 80%.The experiment transitions from six initial models to five unseen additions.
  • OOD Generalization: On OOD MMLU-Pro queries, R2-ROUTER achieves AUDC 0.71 versus 0.67 for CARROT-L, outperforming all baselines under a STEM-to-non-STEM shift.This result supports cost-efficiency and robustness under distributional shift.
  • Ablation Study: With K = 4 anchor points, R2-ROUTER already outperforms MIRT and CARROT-L, while QNC approaches 0.12 with 6 to 8 points.Piecewise linear interpolation approximates the continuous quality–cost curve using discrete anchor budgets.
  • Robustness Studies: R2-ROUTER remains effective with MiniLM embeddings, an LGBM predictor, and DeepSeek-V3.1 as the evaluation judge.With MiniLM it reaches QNC 0.32; with LGBM it reaches AUDC 0.80 and QNC 0.29.
  • Ablation Study: Length-constrained prompting alone does not match R2-ROUTER because reactive routers still use static cost estimates and exclude efficient configurations involving powerful LLMs.The comparison isolates routing logic from merely prompting models to be concise.

7. Conclusion

The paper identifies fixed quality–cost assumptions as a limitation of existing routers and introduces curve-based routing with R2-ROUTER and R2-BENCH. Experiments report lower cost, strong generalization, and compatibility with existing routers.

  • Existing routers overlook quality variation with output length, so R2-ROUTER searches over LLM quality–cost curves instead of fixed points.
  • R2-BENCH captures LLM behavior across diverse token budgets, enabling routing models to learn quality changes with output length.
  • R2-ROUTER achieves state-of-the-art performance at 4−5× lower cost, generalizes to new LLMs and OOD queries, and integrates as a plug-in module.

Impact Statement

The work targets more cost-efficient LLM inference through intelligent routing. It may improve access to high-quality LLM responses and reduce AI systems’ environmental footprint.

  • The work aims to improve LLM inference cost-efficiency through intelligent routing.
  • Reduced computational cost may make LLM capabilities more accessible and reduce AI systems’ environmental footprint.

A. Effectiveness of Length-Constrained Instructions

Length-constrained instructions are followed reliably by large models but much less reliably by small models at tight budgets. R2-ROUTER focuses on exploiting controllable large-model configurations, and training on actual constrained responses helps account for compliance behavior.

  • Large models exceed 82% compliance at a 10-token budget and exceed 97% at budgets of at least 100 tokens.Compliance is measured as responses whose actual length is at most 1.1× the budget.
  • Models below 4B parameters have only 3%–21% compliance at the 10-token budget.
  • R2-ROUTER targets powerful LLMs with constrained budgets because small models are already inexpensive and offer little additional benefit from tighter outputs.
  • Training on actual constrained responses lets R2-ROUTER learn compliance behavior and reflect excess output costs in the learned quality–cost curve.

B. Details of Experiments

R2-BENCH combines diverse routing tasks with a pool of 15 LLMs and evaluates quality–cost behavior across output budgets. The experiments primarily use discrete routing over 16 anchor budgets without interpolation.

  • R2-BENCH integrates queries from 6 widely-used benchmarks spanning 20 categories.
  • The benchmark includes professional multiple-choice, general knowledge, mathematical problem-solving, graduate-level science, and multi-step reasoning tasks.
  • The LLM pool spans 15 models ranging from 0.6B to 235B parameters, including general-purpose and domain-specific models.
  • Pricing values were retrieved from OpenRouter in January 2026 and are subject to change.
  • Unless otherwise stated, experiments use discrete routing over K = 16 anchor budgets without interpolation.

C.1. Evaluation on RouterArena

RouterArena evaluates R2-ROUTER across broad tasks using exact-match scoring, while additional analysis explains how its budget controls reasoning and how interpolation trades accuracy for data efficiency.

  • Evaluation on RouterArena: RouterArena covers 8,400 queries across 7 categories and 5 dimensions, using exact-match scoring without an LLM judge.
  • Evaluation on RouterArena: R2-ROUTER leads in every RouterArena category under exact-match scoring.
  • Reasoning capacity: For reasoning LLMs, the controllable thinking-token budget governs reasoning capacity, whereas non-reasoning models express reasoning directly in output tokens.
  • Prompting and budgets: Budget instructions can be combined with model-specific system prompts because prompts control style and format while the budget guides output length.
  • Interpolation: PCHIP achieves lower prediction error and better routing QNC than PLI at non-anchor budgets with K = 8 anchors.
  • Interpolation: Interpolation is optional: main-text results use K = 16 discrete anchor budgets, while interpolation reduces data-collection cost at the expense of approximation error in non-convex regions.
Loading 2602.02823v2…