PaperlayerView exact revision on arXiv

Source-linked AI summary

SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing

Vasilis Perifanis, Nikolaos Pavlidis, Symeon Symeonidis

arXiv:2609.34736v1cs.AIcs.CLcs.LG

TL;DR

LLM routers often use representations that do not explicitly state what a query requires. SeLMRoute extracts candidate-independent probabilistic semantic evidence, then learns candidate performance and applies routing objectives; it reports competitive benchmark results and positive performance gains in a separate cost-aware setting.

  • Problem

    Many routing approaches use query embeddings, model representations, preferences, or clusters, while the representation often does not explicitly describe what a query requires.

    Source: §1 · p. 1

  • Method

    SeLMRoute retains probabilistic judgments about interpretable query requirements as candidate-independent semantic evidence, then learns candidate performance and applies routing objectives.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    Across five grouped splits, SeLMRoute achieves 72.08 ± 0.45% AvgAcc and a mean PerfGain of 2.66% in the performance-cost setting.

    Source: §5 · p. 17

  • Takeaways & Limitations

    The same semantic state can support different candidate pools and deployment objectives without changing the query representation.

    Source: §1 · p. 2

    The semantic evidence is inspectable, but it does not fully explain the downstream nonlinear regressor's decisions.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Large language model (LLM) routing aims to select the most suitable model for each incoming query. Most existing routers learn this decision directly from query embeddings, model representations, preference data, or clusters of similar examples. Such approaches can be effective, yet the representation used for routing rarely states what a query actually requires. We introduce SeLMRoute, a routing framework that separates the extraction of candidate-independent semantic evidence from the learning of candidate performance and the application of deployment objectives. A decision model first evaluates a set of interpretable questions about the query, such as its reasoning requirements and use of external knowledge, with each judgment retained as a probability distribution. The resulting probabilistic semantic state is used by a lightweight supervised router to estimate candidate model performance. Routing objectives are applied after performance estimation, which allows the same semantic state to support performance-oriented and cost-aware decisions. On the LLMRouterBench (15 datasets, 20 candidate models, 11,481 queries), SeLMRoute achieves an average accuracy of $72.08\% \pm 0.45$, while grouped five-fold out-of-fold evaluation reaches $72.64\%$, compared with $69.23\%$ for the strongest fixed candidate. The representation achieves the highest mean performance among the evaluated semantic, dense, lexical, and domain-level representations. In a separate 13-model performance-cost setting, SeLMRoute improves performance in all five grouped splits, with a mean PerfGain of $2.66\%$. Our code is available at https://github.com/Indigma-Innovations/SeLMRoute.

1 Introduction

LLM routing must account for requests with different requirements and models with different capabilities, but common representations often do not make those requirements explicit. SeLMRoute uses probabilistic semantic evidence before learning model performance, and reports competitive routing results across its evaluations.

1 Introduction

  • The framework retains probability distributions over sixteen semantic dimensions as a 40-dimensional representation for a separate learner to estimate candidate performance.The dimensions cover requirements such as reasoning type, knowledge needs, task structure, ambiguity, and exactness.Source: §1 · p. 2
  • Semantic evidence makes task requirements inspectable, but it does not provide a causal explanation of every decision made by the nonlinear learner.The evidence can expose properties such as code reasoning, exactness, constraint density, and dependence on current information.Source: §1 · p. 2
  • 72.08%±0.45 AvgAcc across five grouped splits is competitive with established routers, while grouped OOF evaluation reaches 72.64 versus 69.23 for the best fixed candidate.Published-router results use their own benchmark split protocol, so they are external comparisons.Source: §1 · p. 2
  • 72.08%±0.45 AvgAcc makes raw probability mass the strongest mean representation evaluated, ahead of larger semantic, dense, lexical, and domain-level alternatives.The experiments also compare hard semantic decisions, GTE-Qwen2 embeddings, coarse domain labels, and TF-IDF features.Source: §1 · p. 2
  • In a separate cost-aware evaluation, SeLMRoute improves performance in all five grouped splits, with mean PerfGain of 2.66%, but establishes no positive monetary savings.The paper also evaluates grouped in-distribution splits, distribution shift, alternative learners, reduced probe sets, an open-weight decision model, and direct routing.Source: §1 · p. 2
  • SeLMRoute separates candidate-independent semantic evidence extraction, candidate performance learning, and configurable routing objectives.The same semantic representation can support performance-oriented or budget-sensitive decisions without changing the representation.Source: §1 · p. 2

2 Related Work

Prior routing work varies in how it represents queries and candidate models, while benchmark results show that leading approaches can be close. SeLMRoute instead represents query requirements as explicit probabilistic evidence before learning candidate performance.

Routing among language models

  • External LLM routing selects among complete candidate systems, unlike mixture-of-experts routing among internal networks; prior systems also study cost-sensitive cascades and strong-versus-weak routing.SeLMRoute places its main contribution in the semantic representation that precedes performance or cost objectives.Source: §2 · p. 3

Query and model representations

  • Representation approaches include contrastive query-model embeddings, capability vectors, heterogeneous graphs, aptitude-style instructions, and in-context model representations.Avengers instead clusters embedded queries and scores candidate models within clusters.Source: §2 · p. 3
  • SeLMRoute assigns declared semantic meanings to query-vector dimensions and retains uncertainty, separating what a request appears to require from which model performs well under those requirements.Its features can describe code reasoning, factual recall, exactness, ambiguity, or constraint density.Source: §2 · p. 3

Hard capability labels and probabilistic evidence

  • For score, categorical, and binary judgments, SeLMRoute retains probability mass over possible levels or alternatives rather than forcing each judgment into a hard label.JEV supplies typed decisions and probabilities, while observed candidate outcomes train the final performance mapping.Source: §2 · pp. 3–4

Benchmarking model routers

  • LLMRouterBench compares routers across common pools, datasets, and metrics; SeLMRoute adds grouped partitions for duplicate inputs and selects cost-aware operating points on validation data.Published systems are treated as external references, with paired improvement claims limited to systems evaluated under the shared grouped protocol.Source: §2 · p. 4

3 The SeLMRoute Framework

SeLMRoute separates query semantics, candidate-performance estimation, and deployment policy: an interpretable probabilistic state feeds a supervised predictor before a configurable objective selects a model.

3.1 Problem Formulation

  • The router factorizes query handling into semantic extraction, candidate-performance prediction, and user-selected routing objectives, keeping deployment preferences separate from learned performance.Source: §3.1 · p. 4, §3 · p. 4

3.2 Architecture and Semantic Evidence Extraction

  • A decision model answers typed, candidate-agnostic questions once per query; compatible probabilities form reusable fixed-length state across candidate pools and objectives.Source: §3.2 · p. 5

3.3 Interpretable and Hard Semantic Judgments

  • Probability mass preserves uncertainty and distinctions between judgments with the same most-likely answer, unlike a 16-feature hard state.Source: §3.3 · pp. 5–6

3.4 Learning Model Performance

  • A supervised performance model maps semantic state to expected score for each candidate; changing pools reuses encoded queries but requires empirical evidence for new models.Source: §3.4 · pp. 6–7

3.5 User-Configurable Routing Objectives

  • A configurable objective selects among predicted candidate outcomes, including quality-cost trade-offs and policy-based candidate restrictions.Source: §3.5 · p. 7

4 Experiments

The experiments test whether explicit semantic evidence supports competitive routing, how its representation matters, and whether it transfers across settings and supports cost-aware routing. The setup uses duplicate-query-safe grouped evaluation across a multi-dataset, multi-model benchmark.

4 Experiments

  • The experiments examine routing competitiveness, semantic-state representation, separation from direct decision-model routing, transfer across settings, and performance-cost routing.Source: §4 · p. 7

4.1 Experimental Setup

  • The main benchmark covers 15 datasets and 20 candidate models, with 11,481 query instances and 229,620 candidate outcomes.The frozen bundle has one more MMLU-Pro record than the published benchmark count; internal comparisons use the same 11,481-instance support.Source: §4.1 · p. 8
  • Representations include semantic probability features, dense embeddings, TF-IDF, and broad domain labels, with DomainOnly ignoring query text and within-domain query differences.The setup also evaluates JEV and an open-weight Laya extractor under the same probe schema and downstream routing procedure.Source: §4.1 · p. 8
  • Grouping by dataset and normalized query keeps repeated inputs in one partition, preventing identical queries from appearing in both training and testing.The benchmark contains 11,423 unique groups; 47 groups contain repeated queries, accounting for 58 additional instances.Source: §4.1 · p. 8
  • Evaluation uses five grouped 70/30 train-test splits and five-fold grouped out-of-fold predictions covering every benchmark instance.The folds keep each group together, and the OOF predictions support paired representation, decision-model, and probe-reduction comparisons.Source: §4.1 · pp. 8–9
  • The evaluation follows LLMRouterBench accuracy metrics and uses grouped bootstrap intervals and paired permutation tests for statistical comparisons.Resampling preserves repeated-query groups and dataset contributions to macro-average accuracy.Source: §4.1 · p. 9

4.2 Main Performance-Oriented Routing

SeLMRoute's ProbabilityMass representation performs strongly in the grouped performance-oriented evaluation and exceeds the reported results of several published routers. The published comparison uses different evaluation protocols, so it is contextual rather than paired.

4.2 Main Performance-Oriented Routing

  • ProbabilityMass records the highest mean AvgAcc of 72.08±0.45, the largest Gain@B of 5.72%, and the smallest Gap@O of 22.00% among evaluated representations.Source: §4.2 · p. 9
  • Across five grouped splits, Best Single reaches 68.81 ± 0.34 AvgAcc, compared with 73.94 ± 0.51 for Dataset Oracle and 91.91 ± 0.43 for Instance Oracle.ProbabilityMass is 3.27 percentage points above Best Single and 1.86 points below Dataset Oracle; its gap to Instance Oracle indicates remaining instance-level opportunities.Source: §4.2 · p. 9
  • 72.08 ± 0.45 AvgAcc places SeLMRoute above the reported results of RouterDC, GraphRouter, EmbedLLM, MODEL-SAT, and Avengers.Source: §4.2 · pp. 9–10, Table 4 · p. 10
  • Published-router values use a different prompt-grouped protocol and an 11,480-prompt benchmark, so they are not used for paired statistical claims against SeLMRoute.SeLMRoute uses the duplicate-query-safe grouped protocol and an 11,481-record frozen bundle.Source: §4.2 · p. 10

4.3 Probabilistic, Full, and Hard Semantic States

The ablation compares probability distributions, expanded semantic features, and discrete judgments. ProbabilityMass leads in the five grouped train-test splits, while the grouped OOF comparison does not establish a significant advantage over Hard.

4.3 Probabilistic, Full, and Hard Semantic States

  • 72.64% AvgAcc for ProbabilityMass versus 71.81% for Hard yields a 0.83-point difference, but the paired OOF test is not significant at the 5% level.The 95% bootstrap interval includes zero and the permutation test reports p = 0.1099.Source: §4.3 · p. 10
  • ProbabilityMass achieves the highest AvgAcc across five grouped splits with 40 features; Full uses 88 features and Hard reduces each probe to a discrete judgment.Full adds derived quantities, while Hard retains only the most likely outcome.Source: Figure 2 · p. 10, §4.3 · p. 10
  • In grouped five-fold OOF evaluation, ProbabilityMass reached AvgAcc 72.64% versus Hard’s 71.81%, a 0.83-point difference, but the clustered 95% bootstrap interval [−0.174, 1.837] included zero and paired permutation p = 0.1099; the difference was not significant at 5%.Source: §4.3 · p. 10

4.4 Sensitivity to the Performance Learner

With the ProbabilityMass semantic state fixed, CatBoost outperforms the tested performance learners; a separate comparison favors semantic evidence extraction followed by performance prediction over direct model selection.

4.4 Sensitivity to the Performance Learner

  • 72.08 ± 0.45 AvgAcc for CatBoost leads the tested performance learners with ProbabilityMass fixed.Random Forest reaches 71.15 ± 0.46 AvgAcc; MLP, Ridge, and OLS score lower.Source: §4.4 · pp. 10–11, Figure 3 · p. 11

4.5 Does the Intermediate Semantic State Matter?

  • 72.57% AvgAcc for SeLMRoute ProbabilityMass exceeds JEV-Direct’s 68.78% by 3.79 percentage points (95% interval [2.69, 4.95]).JEV-Direct selects from anonymized candidate profiles without an intermediate semantic representation or separate performance regressor.Source: §4.5 · p. 11, Figure 4 · p. 11
  • The comparison supports using reusable semantic evidence and a separate learner to map that evidence to candidate performance.The same semantic state can be retained while changing the downstream performance learner.Source: §4.5 · p. 11

4.6 Transfer Across Decision Models

The semantic-state architecture works with an open-weight decision model, while reducing the probe set lowers extraction overhead but does not establish noninferiority. Probe ablations highlight structural and current-information requirements, while negative marginal drops need cautious interpretation.

4.6 Transfer Across Decision Models

  • 70.53% AvgAcc for Laya-SeLMRoute remains above Best Single at 69.23%, but below JEV-SeLMRoute at 72.64%.The JEV–Laya macro difference is 2.10 percentage points (95% clustered bootstrap interval [0.940, 3.255]; p = 0.00035).Source: §4.6 · pp. 11–12

4.7 Reducing the Semantic State

  • The twelve-probe configuration reaches 72.49% AvgAcc versus 72.64% for sixteen probes, but noninferiority within 0.5 percentage points is not established.The one-sided 95% lower confidence bound is −0.783 percentage points; the reduced state also cuts semantic input tokens by approximately 34% and p95 extraction latency by approximately 18%.Source: §4.7 · p. 12

4.8 Which Semantic Judgments Matter?

  • Removing constraint density, context integration, or current information produces the strongest reductions in AvgAcc.The first two describe structural requirements spanning benchmark domains; current information distinguishes requests whose correctness depends on recent information.Source: §4.8 · p. 12, Table 5 · p. 13
  • Slight gains after removing some probes do not show those concepts are irrelevant, because leave-one-out effects measure marginal value with other signals retained.Small negative drops may also reflect estimation noise or overfitting; the largest negative change is 0.19%.Source: §4.8 · pp. 12–13

4.9 Generalization Under Distribution Shift

Semantic representations remain competitive when datasets are held out, but a wholly unseen task domain is a harder transfer setting. Results distinguish transfer across benchmark boundaries from extrapolating candidate performance into a domain absent from training.

Dataset-OOD

  • 67.02% AvgAcc for ProbabilityMass is close to Full at 67.27% and GTE-Qwen2 at 67.29% under Dataset-OOD; the ordering shows no clear representation advantage.All representations score lower than in the in-distribution evaluation.Source: §4.9 · p. 13, Figure 6 · p. 14

Domain-OOD

  • 67.31% AvgAcc for GTE-Qwen2 leads Domain-OOD, while ProbabilityMass reaches 65.66%.DomainOnly reaches 65.64%, TF-IDF 65.62%, and Full 65.44% across the five held-out domains.Source: §4.9 · p. 13, Figure 6 · p. 14

Generalization under distribution shift

  • Dataset-OOD can transfer when related task characteristics remain in training, whereas Domain-OOD leaves the learner without evidence about candidate behavior in the excluded domain.The semantic extractor can still describe a query from that domain, but this does not supply the missing candidate-performance relationships.Source: §4.9 · pp. 13–14

4.10 Semantic Uncertainty and Selective Routing

Semantic entropy ranks difficult routing cases slightly better than the downstream margin in grouped in-distribution evaluation, but calibrated rejection yields no meaningful accuracy gain on unseen domains. In the separate performance-cost setting, SeLMRoute improves performance across splits, while the strict protocol finds no positive cost savings.

4.10 Semantic Uncertainty and Selective Routing

  • Semantic entropy has an AURC of 0.0901 versus 0.0974 for the downstream router margin, ranking in-distribution routing errors more effectively.Lower AURC indicates errors are concentrated more effectively among queries rejected first.Source: §4.10 · p. 14
  • 65.38 ± 0.60% AvgAcc at 96.82 ± 0.92% coverage under Domain-OOD is close to routing all queries at 65.32 ± 0.60%.The entropy threshold is calibrated on in-distribution data and held fixed for each unseen domain.Source: §4.10 · p. 14
  • Semantic entropy measures uncertainty in query characterization, not how well candidate behavior is supported in an unseen region.A query may receive a confident semantic description even when training data provide little evidence about candidate performance there.Source: §4.10 · p. 14

4.11 Performance-Cost Routing

  • The performance point maximizes validation AvgAcc, while the cost point is the least-cost validation configuration that meets GPT-5’s validation accuracy.The selected operating points are evaluated on held-out test data; strict CostSave requires the cost point also to meet the test accuracy condition.Source: §4.11 · p. 15
  • PerfGain is positive in all five grouped splits, averaging +2.66 ± 1.85% against GPT-5, while positive strict CostSave occurs in none.The only validation-selected configurations meeting the held-out accuracy requirement have CostSave values of −0.10% and −1.58%.Source: §4.11 · p. 15, Table 6 · p. 15

4.12 Dollar Cost of Semantic Extraction

The section estimates the API cost of generating SeLMRoute’s semantic state and notes that latency is a separate operational overhead.

4.12 Dollar Cost of Semantic Extraction

  • $0.775 total, or $0.0675 per 1,000 queries, is the estimated cost of extracting the full sixteen-probe state across 11,481 queries.The estimate uses 18,446,536 input tokens at the evaluation-time JEV rate of $0.042 per million input tokens.Source: §4.12 · p. 16
  • $0.512, or approximately $0.0446 per 1,000 queries, is the estimated cost of Lite-12 extraction, about 33.9% below the full configuration.Lite-12 uses an average of 1,062.7 input tokens per query and 12,200,872 tokens across the same benchmark.Source: §4.12 · p. 16
  • The estimated extraction cost is reported separately because token measurements and cost-aware routing use different query populations.Adding these expenditures directly would combine measurements from the 11,481-query performance-oriented benchmark and the 12,446-query cost-aware setting.Source: §4.12 · p. 16
  • The estimates indicate that semantic-extraction charges are small relative to recorded flagship-model inference costs, and fewer probes lower them further.The monetary estimates use the evaluation-time JEV price.Source: §4.12 · p. 16
  • The monetary estimate excludes latency from the additional decision-model invocation, which may affect suitability for time-sensitive deployments.Latency is examined separately in Section 4.13.Source: §4.12 · p. 16

4.13 System Overhead

Semantic extraction accounts for most of SeLMRoute’s additional latency, while downstream CatBoost prediction is comparatively fast. The evaluated backends and reduced-probe configuration illustrate latency and accuracy trade-offs, but the backend measurements are not hardware-normalized comparisons.

4.13 System Overhead

  • Approximately 350 ms mean and 481 ms p95 extraction latency are observed for the full sixteen-probe JEV configuration, including remote API and network overhead.Its mean input is approximately 1,607 tokens per query; the latency is for the extraction stage, not isolated decision-model execution.Source: §4.13 · p. 16
  • Laya records 103 ms mean and 206 ms p95 extraction latency, but its local measurements are not directly comparable with JEV’s remote-API measurements.Under the shared semantic schema, JEV-SeLMRoute achieves higher routing accuracy than Laya-SeLMRoute.Source: §4.13 · p. 16
  • At batch size one, CatBoost prediction takes 1.23 ms per query, falling to 0.043 ms at batch size 32 and 0.011 ms at batch size 256.All three downstream prediction configurations are substantially faster than semantic extraction.Source: §4.13 · p. 16
  • Reducing to twelve probes cuts input tokens by 34% and p95 extraction latency by 18%, while grouped OOF accuracy decreases by 0.145 percentage points.The reduction in p95 latency is from approximately 481 ms to 394 ms; statistical noninferiority to the full configuration is not established.Source: §4.13 · p. 16
  • The authors identify semantic extraction as the main additional latency cost and point to adaptive probe execution and faster or specialized decision backends as efficiency directions.Although JEV extraction’s estimated monetary cost is low under evaluated API pricing, its latency matters more for deployments requiring short response times.Source: §4.13 · pp. 16–17
  • Figure 7 presents grouped five-fold OOF routing accuracy alongside observed extraction latency for JEV-SeLMRoute and Laya-SeLMRoute.JEV latency includes remote API and network overhead, while Laya is measured locally; the measurements are not hardware-normalized model benchmarks.Source: §4.13 · p. 16, Figure 7 · p. 17

5 Conclusions and Future Work

The paper concludes that SeLMRoute separates interpretable probabilistic semantic evidence from performance learning and routing objectives, with competitive benchmark performance and a positive mean PerfGain in the performance-cost setting. It also identifies generalization, extraction latency, monetary savings, and candidate cold start as open problems, and outlines directions for addressing them.

5 Conclusions and Future Work

  • Across five grouped splits, SeLMRoute achieves 72.08 ± 0.45% AvgAcc and a mean PerfGain of 2.66% in the performance-cost setting, though reliable monetary savings are not established.The semantic interface supports different decision models and candidate pools, while preserving probability mass offers a compact alternative to larger representations.Source: §5 · p. 17
  • The authors identify difficult generalization to entirely unseen domains, measurable semantic-extraction latency, and the lack of guaranteed monetary savings as open problems.When a new candidate LLM is added, previously extracted semantic states remain reusable, but the performance learner requires candidate-specific calibration.Source: §5 · p. 17
  • Future work will investigate decision models specialized for semantic routing, probability calibration, joint treatment of semantic and routing uncertainty, and adaptive probe selection.The proposed studies include fine-tuning an open decision model on SeLMRoute’s typed semantic-analysis task.Source: §5 · p. 17
  • The authors also plan active calibration for new candidates, online adaptation as capabilities or prices change, and objectives jointly accounting for quality, inference cost, extraction cost, and latency.Active calibration would select a small, informative set of semantic regions for evaluating newly introduced models.Source: §5 · p. 17

Related papers

Loading 2609.34736v1…