PaperlayerView exact revision on arXiv

Source-linked AI summary

SCX Router: Streaming Zero-Shot Model Selection with a Decoder-KV Classifier and a Real-World Task Ontology

Ihor Stepanov, Aleksandr Smechov, Mykhailo Shtopko, Dmytro Vodianytskyi, Oleksandr Lukashov

arXiv:2609.02292v1cs.AIcs.CL

TL;DR

SCX Router addresses the difficulty of selecting among heterogeneous LLM endpoints as task requirements, costs, latency, and constraints vary. It uses a decoder-KV, dynamic-label classifier with auxiliary task signals and a synthetic task ontology, while leaving deployment policies explicit. On a selected 1,000-task subset, it reaches top-1 score 0.707 versus 0.696 for the strongest fixed endpoint, but gains depend on candidate disagreement and evaluation coverage.

  • Problem

    Heterogeneous LLM endpoints differ in quality, price, latency, context, tools, and operational constraints, making per-task model selection difficult.

  • Method

    SCX Router uses a decoder-KV GLiClass classifier to score dynamic natural-language model labels and predict task attributes, with eligibility and deployment policies handled separately.

  • Results

    0.707 versus 0.696: on the selected 1,000-task evaluation subset, SCX Router exceeds the strongest fixed endpoint at top-1, with mixed gains across depths and benchmarks.

  • Takeaways & Limitations

    The router provides a compact performance signal for constrained policies over evolving endpoint rosters and realistic tool-using or agentic workflows.

  • Takeaways & Limitations

    The reported end-to-end evidence uses a selected subset lacking uncertainty intervals, repeated seeds, endpoint versions, and held-out fixed-policy selection, while expanded endpoints have unequal coverage.

Abstract

from arXiv · show

The rapid proliferation of large language models (LLMs) and the growing diversity of their applications presents a unique optimization opportunity: selecting the right model for the task, while optimizing for speed, cost, and quality at a per-task level. However, inference endpoints can vary widely in quality, price, latency, context support, tool use, domain expertise, and reasoning behavior. This heterogeneity makes manual heuristics difficult to maintain and unlikely to achieve consistently favorable speed--cost--quality trade-offs on their own. We introduce \router{}, a lightweight GLiClass-based router that assigns a suitability score to each inference-time model label without autoregressive generation. The released 0.6B-parameter checkpoint combines a Qwen3 decoder with a shallow bidirectional scorer. Its decoder-KV execution path preserves a text-only key--value cache across a session, encodes only new dialogue turns, and evaluates transient candidate-label tokens without adding them to the persistent cache. The same checkpoint also predicts task type, difficulty, reasoning mode, and expected output length, and supports custom zero-shot labels. For task generation, we construct a task ontology with 23 families, 115 task types, 345 routable subtypes, 1,173 synthetic examples, and an orthogonal axis of 30 domains. Using this structure, we generate 150,000 verifier-scored tasks and 15,000 open-ended tasks. We then train the Qwen3 decoder on these tasks, while explicitly separating learned request prediction from per-task policies for attributes such as eligibility, cost, cache reuse, safety, and sovereignty. Across six LiveBench subsets, the router outperforms the mean candidate; on the selected 1,000-task subset, it achieves an aggregate top-1 score of 0.707 versus 0.696 for the strongest fixed model, with benchmark-dependent gains.

1 Introduction

SCX Router addresses dynamic, heterogeneous model selection with a streaming zero-shot classifier that scores natural-language candidate labels without generation. It combines reusable conversational context, multi-signal predictions, explicit deployment policies, and a task ontology for realistic workflows.

  • Streaming zero-shot router: SCX Router scores natural-language candidate labels in a non-generative forward pass, supporting changing model rosters and custom zero-shot labels.The same interface also handles task taxonomies, difficulty, reasoning modes, output-length buckets, and safety signals.
  • Streaming zero-shot router: Decoder-KV execution reuses cached request context across dialogue turns while scoring transient candidate labels without adding them to the persistent cache.This creates a streaming classifier rather than a second conversational generator.
  • Policy separation: Learned suitability prediction remains separate from deterministic policies that enforce eligibility, safety, privacy, residency, tools, cost, latency, and cache constraints.The decision layer can exclude ineligible endpoints before trading predicted performance against operational factors.
  • Task ontology and evaluation: The ontology contains 23 task families, 115 task types, 345 routable subtypes, 1,173 synthetic examples, and an orthogonal 30-domain axis.It structures 150,000 verifier-scored and 15,000 open-ended synthetic tasks with workflow structure.
  • Task ontology and evaluation: The evaluation distinguishes mean-candidate and fixed-model baselines while preserving observation masks for unequal endpoint coverage.Missing outcomes are not treated as failures in the expanded collection.
  • Positioning: Unlike prompted generative routers, SCX Router keeps dynamic labels and cacheable causal processing while producing discriminative outputs.Its design occupies a middle point between cheap label encoders, interaction-heavy cross-encoders, and generative routers.

3 Task Formulation

The task formulation separates observed-outcome supervision from policy-based routing. It masks missing evaluations, restricts candidates using hard constraints, and optimizes an objective that can combine suitability, performance, cost, latency, and cache reuse.

  • Routing supervision: Routing targets are derived from benchmark-normalized scores and observation indicators, selecting models tied for the highest observed outcome within an optional tolerance.This avoids imposing one universal success threshold across heterogeneous evaluators.
  • Coverage handling: Unobserved task–model pairs are treated as missing data rather than failures and must be masked or restricted to a paired intersection.The eleven-endpoint collection therefore does not support 11-way comparative accuracy without suitable coverage handling.
  • Policy decision: The policy first restricts candidates to those satisfying context, tool, modality, region, privacy, and safety constraints.Only eligible candidates Et ⊆ Mt enter the final utility comparison.
  • Policy decision: The final utility can combine request-specific scores, empirical performance profiles, expected incremental cost, latency, and cache reuse.The learned classifier therefore supplies a signal to a broader deployment policy rather than determining the route alone.

4 Decoder-KV Router

The decoder-KV router scores natural-language candidate labels through a non-generative GLiClass architecture while reusing text-only session state. It separates learned request signals from deployment policy and supports dynamic rosters without contaminating dialogue cache state.

  • Architecture: The approximately 0.6B-parameter checkpoint combines a Qwen3-0.6B causal backbone with a bidirectional scorer and shared MLP.The scorer extracts decoder-conditioned text and label-marker representations before producing per-label scores.
  • Deployment Policy: A separate application policy applies hard eligibility constraints and trades predicted performance against cost, latency, and cache state.The classifier returns request signals; endpoint selection remains outside the learned score.
  • Streaming Execution: Persistent dialogue-context tokens update the text-only cache, while label-stage keys and values are discarded after scoring.This preserves dialogue state while allowing repeated classification with changing label rosters.
  • Streaming Execution: Cache reuse amortizes request-side representation but does not make routing cost independent of retained history or roster size.The transient label suffix still attends to retained history, so latency depends on cache length and label-token count.
  • Architecture: Candidate labels are transient inputs whose marker states receive independent suitability scores through a shared per-label MLP.Natural-language labels can change at inference time, enabling variable candidate rosters.

5 Task Ontology

The task ontology separates stable intent from domain and routing dimensions to represent diverse application contexts without an unstable Cartesian label vocabulary. Its hierarchy provides routable granularity while cross-cutting dimensions capture reasoning, risk, interaction, and other constraints.

  • Intent Hierarchy: The hierarchy contains 23 families, 115 task types, and 345 routable subtypes, with five task types and 15 subtypes per family.It also includes 1,173 synthetic example specifications for coverage and downstream task generation.
  • Intent, Domain, and Cross-Cutting Dimensions: Eight cross-cutting dimensions represent reasoning level, risk, freshness, interaction mode, modality, output mode, route kind, and instruction complexity.Multi-turn and instruction-following behavior are represented as interaction and constraint dimensions rather than top-level intent classes.
  • Label Boundaries: The taxonomy distinguishes analysis, reasoning, problem solving, comparison, evaluation, critique, verification, retrieval, question answering, and fact checking by operation.These boundaries address confusion among conceptually overlapping labels in the released 28-way task-type evaluation.
  • Intent, Domain, and Cross-Cutting Dimensions: The ontology models intent independently from 30 domains, instantiating domain-specific examples only when needed.Composing 345 subtypes with 30 domains yields 10,350 initial design cells without requiring all cells to be populated.

6 Data Sources and Supervision

Routing supervision combines benchmark-derived outcomes with synthetic tasks structured by the ontology and real-world workflow context. The data pipeline preserves observation status and defines difficulty relative to the evaluated candidate roster.

  • Data Sources: Routing supervision comes from benchmark-derived prompts and synthetic tasks generated from the task ontology.The synthetic path represents workflow structure such as dialogue state, tools, files, repository context, and acceptance criteria rather than production-log provenance.
  • Routing Supervision: Benchmark prompts are evaluated across candidate models, retaining each endpoint’s score and observation indicator for routing-target construction.Endpoint names or descriptions do not determine the target.
  • Routing Supervision: Difficulty is assigned after model evaluation from relative candidate-roster performance, so it depends on the roster and success rule.Tasks solved by many models are easier, while tasks solved by fewer models are harder.
  • Training Data: The focused training mixture contains 65,099 classification records spanning routing, difficulty, task type, output length, reasoning mode, and propositional logic.The broader GLiClass mixture contains 524,035 records; these are classification-record counts, not unique benchmark-prompt counts.
  • Synthetic Task Generation: 150,000 generated tasks use deterministic, executable, or environment-based verification, while 15,000 open-ended tasks are judged by gpt5.6-sol.Together, the pools contain 165,000 synthetic tasks with retained evaluation provenance.
  • Outcome Coverage: Unequal endpoint coverage means missing evaluations are not failures and prevents direct comparison of raw model averages across different task mixtures.The expanded evaluation uses eleven endpoint identifiers, but a complete 165,000×11 outcome matrix is not implied.

7 Training

The released checkpoint supports multi-label routing and auxiliary classification, while separating learned predictions from deployment policies and distinguishing evaluated, implemented, and proposed routing patterns.

  • Training Configuration: The released decoder-KV model supports multi-label classification with binary targets, configurable focal modulation, and shuffled label order.Only labels in all_labels are tokenized into the suffix; exact focal arguments and complete launch commands are unavailable.
  • Training Configuration: 0.9273 flattened weighted binary accuracy and 0.9241 F1 measure label decisions, not example-level exact-match routing accuracy.The paper therefore treats routing-family and downstream results as more relevant evidence.
  • Routing Patterns: The classifier supports direct, attribute-mediated, hybrid, and hierarchical routing after restricting candidates to hard eligibility constraints.These constraints include context, modality, tools, privacy, residency, and safety.
  • Routing Patterns: Direct routing scores eligible endpoint labels, while attribute-mediated routing predicts stable attributes and consults replaceable performance profiles.The released eight-endpoint checkpoint implements and evaluates the direct path; the profile path is implemented without end-to-end results.
  • Routing Patterns: Hybrid routing combines direct and profile evidence, whereas proposed probabilistic, cascade, portfolio, and agentic variants require additional evaluation or deployment resources.Hierarchical routing can assign planner, executor, verifier, and synthesizer roles, but it has not been evaluated end to end.

9 Cache-Aware Routing Policy

The cache-aware policy prices only the next action, accounts for cache reuse and switching costs, and combines normalized performance and cost signals only after hard eligibility filtering.

  • Cost Model: Equation 19 prices the next action using cached history, new input, and expected output tokens with model-specific input and output rates.Previously billed tokens are excluded from the calculation.
  • Cost Model: Switching models replays conversation history at the new candidate’s full input rate, making long cache-warm conversations naturally sticky.The effective history rate mixes cached-input and full-input pricing according to cache-reuse probability for the current model.
  • Decision Policy: 0.60 suitability, 0.15 task fit, 0.15 difficulty fit, and 0.10 reasoning fit are illustrative weights in one configurable multi-signal utility.They are not learned constants or universal defaults.
  • Eligibility: Safety, privacy, residency, tool permissions, and context feasibility remain hard filters that utility cannot compensate for.The policy applies these constraints before performance, cost, latency, cache, and capacity trade-offs.

10 Evaluation

Evaluation covers classification families and downstream routing, showing strong but uneven auxiliary predictions and conditional end-to-end gains over fixed and mean-candidate baselines.

  • Classification: 0.759 explicit per-candidate macro F1 measures released multi-label model-routing classification, while a separate protocol reports 0.805 and is kept separate.The protocols differ, so the values are not interpreted as temporal regression.
  • Model Routing: Precision ranges from 0.739 to 0.840 across eight candidates, while recall ranges from 0.650 to 0.950.The paper argues that threshold tuning should be candidate- and objective-aware.
  • Task Type: Problem solving has F1 0.127, while math-and-reasoning, decision support, information retrieval, and evaluation reach 0.537, 0.599, 0.626, and 0.642.Semantically distinct task classes such as code, QA, summarization, and translation are described as near-perfect.
  • Difficulty: 0.789 macro F1 is obtained for difficulty classification, with most errors occurring between adjacent hard and extra-hard levels.A separate evaluation reports top-1, top-2, and top-3 hit rates of 0.507, 0.748, and 0.873.
  • End-to-End Routing: At k = 1 on the selected 1,000-task subset, fixed/router scores are 0.696/0.707, a gain of +0.012 for routing.At k = 2 the gain is +0.007, while at k = 3 it is −0.013.
  • Evaluation Scope: The selected subset omits five benchmarks and lacks uncertainty intervals, repeated seeds, endpoint versions, and held-out fixed-policy selection.Therefore, the results do not support a global significance claim or downstream comparisons for the unequally covered eleven-endpoint collection.

11 Discussion

SCX Router frames model selection as dynamic-label classification with cache reuse and explicit deployment controls, but its empirical advantage remains conditional and its broader routing designs require further evaluation.

  • Discussion: A compact router can score changing endpoint rosters, reuse conversational context, and expose task, difficulty, reasoning, and output-length signals to deployment policy.Eligibility, safety, price, latency, and cache reuse remain separately controllable.
  • Discussion: The router helps most when candidate models disagree and request semantics predict those differences; a strong fixed model remains competitive when outcomes are similar.Current end-to-end evidence covers direct routing on a selected subset.
  • Future Work: Future evaluations should compare direct, attribute-mediated, hybrid, and hierarchical routing on shared, versioned task strata across broader endpoint capabilities and operating points.The proposed expansion includes model families, sizes, modalities, context lengths, tool use, and price–latency tiers.

12 Conclusion

SCX Router presents compact dynamic-label routing with multi-signal prediction and reports a modest aggregate advantage over fixed selection. Its broader contribution is a foundation for realistic routing evaluation, not a universal leaderboard.

  • 0.707 top-1 score versus 0.696 for the strongest fixed endpoint on the selected 1,000-task evaluation subset.
  • The approximately 0.6B-parameter checkpoint predicts model suitability, task type, difficulty, reasoning mode, and output length over an evolving endpoint roster.
  • The router uses lightweight dynamic-label classification and repeated routing over conversational state without an additional generative decision step.
  • 150,000 verifier-scored tasks and 15,000 gpt5.6-sol-judged tasks broaden routing targets toward realistic applications and agentic workflows.
  • The work provides a foundation rather than a final universal leaderboard, requiring paired outcome matrices and realized quality–cost–latency evaluation to establish operational value.

A Ontology Families

The ontology uses a deliberately regular task-family inventory, with each family receiving the same number of task types, subtypes, and synthetic example specifications across 30 domains.

  • Each ontology family contains five task types, 15 subtypes, and 51 synthetic example specifications.
  • Table 10 presents the complete top-level task-family inventory.
  • The ontology spans 30 domains grouped across eight broad service, science, technology, industry, commerce, environment, and research categories.
Loading 2609.02292v1…