PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillRouter: Skill Routing for LLM Agents at Scale

YanZhao Zheng, ZhenTao Zhang, Chao Ma, YuanQiang Yu, JiHuai Zhu, Yong Wu, Tianze Xu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu

arXiv:2603.22455v5cs.LG

TL;DR

Large skill registries make routing difficult, and name-and-description metadata may omit information needed to identify the right skill among overlapping candidates. The paper tests body access at approximately 80K-skill scale and introduces a compact body-aware retrieve-and-rerank pipeline. SKILLROUTER reaches 74.0% Hit@1 with substantially lower parameter count and latency, while gains transfer to supplementary retrieval and coding-agent task evaluations.

  • Problem

    Existing agent frameworks assume metadata is sufficient for selecting skills, but this assumption is not tested under large open-registry scale and heavy overlap.

  • Method

    SKILLROUTER uses a body-aware bi-encoder to retrieve top-20 candidates and a cross-encoder to rerank them using all skill fields.

  • Results

    74.0% Hit@1 is achieved by the primary 1.2B pipeline, compared with 68.0% for the strongest 16B base pipeline, while routing gains transfer across four coding agents.

  • Takeaways & Limitations

    Body access is a critical routing signal in this benchmark, and a compact body-aware pipeline provides a favorable efficiency–accuracy tradeoff with transfer beyond retrieval metrics.

  • Takeaways & Limitations

    The 75-query core is small, the supplementary benchmark is only a transfer check, and downstream results cover four agents without significance testing.

Abstract

from arXiv · show

Reusable skills let LLM agents package task-specific procedures, tool affordances, and execution guidance into modular building blocks. As skill ecosystems grow to tens of thousands of entries, exposing every skill at inference time becomes infeasible. This creates a skill-routing problem: given a user task, the system must identify relevant skills before downstream planning or execution. Existing agent stacks often rely on progressive disclosure, exposing only skill names and descriptions while hiding the full implementation body. We examine this design choice on a SkillsBench-derived benchmark with approximately 80K candidate skills, targeting the practically important setting of large skill registries with heavy overlap. Across representative dense and reranking baselines on this setting, hiding the skill body causes a 37-44 percentage point drop in routing accuracy. Stronger controls show that the missing signal is body-resident rather than a simple length artifact: body-distilled descriptions recover part of the gap, but remain 7-21 points below direct all-field routing, while a metadata-only encoder trained with the same data remains 14.0 points below its all-field counterpart. Motivated by this finding, we present Skillrouter, a compact 1.2B body-aware retrieve-and-rerank pipeline. Skillrouter achieves 74.0% Hit@1 on our benchmark -- the strongest average top-1 routing performance among the baselines we evaluate -- while using 13$\times$ fewer parameters and running 5.8$\times$ faster than the strongest base pipeline. The ranking gains further generalize to a supplementary benchmark independently constructed from three skill sources. In a complementary end-to-end study across four coding agents, routing gains transfer to improved task success, with larger gains for more capable agents.

1 Introduction

Skill routing becomes necessary as reusable-skill registries grow too large for agents to inspect exhaustively, especially when many skills overlap. This paper evaluates hidden-body routing at approximately 80K-skill scale and presents a compact body-aware alternative.

  • Existing agent systems expose reusable skills as modular procedures, tool knowledge, and execution guidance, but typically present agents only names and descriptions.The routing component can inspect all fields even though downstream agents usually initially see metadata only.
  • 37–44pp drops in routing accuracy show that removing skill bodies sharply harms representative dense and reranking baselines.The study targets a benchmark with approximately 80K skills and heavy overlap.
  • 74.0% Hit@1 is achieved by the primary 1.2B configuration, compared with 68.0% Hit@1 for the strongest 16B base pipeline.The compact system uses 13× fewer parameters and 5.8× lower serving latency.
  • Routing improvements transfer to end-to-end task success across four coding agents, with larger benefits for more capable agents.The paper also evaluates a supplementary benchmark independently constructed from three skill sources.
  • SKILLROUTER is a compact body-aware retrieve-and-rerank pipeline designed for large, homogeneous skill pools.The paper uses standard retrieval and reranking components with setting-specific training adaptations.

2 Problem definition and benchmark

The paper defines routing as retrieving the skills needed for a task from a large pool, where each skill has metadata and a full implementation body. Its benchmark uses expert-curated mappings and separates ordinary large-pool retrieval from confusion among plausible functional alternatives.

  • Skill routing retrieves the required skill set G_q from a large pool S given a task query q.Each skill contains a name, description, and full implementation body, while the downstream agent initially sees only metadata.
  • Hit@1 measures whether any required skill ranks first, while Recall@K and FC@10 characterize shortlist and full-set coverage.The benchmark reports MRR@10, Recall@10/20/50, and FC@10 alongside Hit@1.
  • 75 core queries are retained from 87 SkillsBench tasks after excluding 12 generic-only cases, comprising 24 single-skill and 51 multi-skill queries.The candidate pool contains approximately 80K skills assembled from SkillsBench and a large open-source collection spanning 51 categories.
  • Hard distractors are same-domain, same-technology, or over-generalized alternatives that appear plausible but fail the required function.The Easy/Hard split isolates standard large-pool retrieval from function-level confusion.

3 What signals drive skill selection?

The study finds that skill bodies contain critical routing information omitted by name-and-description metadata. Multiple controls argue against explaining the gap solely through text length or training differences.

  • 37.3pp, 38.7pp, and 44.0pp Hit@1 reductions follow body removal for the 0.6B encoder, 8B encoder, and 8B retrieve-and-rerank baseline, respectively.The collapse occurs across encoder-only retrieval and reranking rather than a single model family.
  • Body-distilled metadata raises Hit@1 from 24.0–27.3% to 52.7–61.3%, but remains 6.7–21.3pp below direct all-field routing.The oracle summaries are query-blind and grounded in each skill’s body.
  • 14.0pp separates the matched metadata-only encoder from its all-field counterpart, which reach 51.3% and 65.3% Hit@1, respectively.Both models use the same 37,979 query pairs and negative recipe, with body withheld from the metadata-only model during training and inference.
  • The proposed pipeline uses name, description, and body content in both retrieval and reranking stages.A bi-encoder retrieves top-20 candidates and a cross-encoder reranks them.
  • 26.9, 50.0, 39.5, and 31.8pp gaps persist across description-length quartiles, including descriptions longer than 35 words.The diagnostic results argue against a pure length explanation, although they are not causal tests.

4 SkillRouter: a compact body-aware routing recipe

SKILLROUTER combines body-aware bi-encoder retrieval with cross-encoder reranking, using training adaptations for near-duplicate skills and fine-grained competition. The design emphasizes efficient use of full skill fields within a narrow candidate set.

  • A bi-encoder retrieves the top-20 candidates from the approximately 80K pool, after which a cross-encoder reranks them using all skill fields.The two stages divide large-pool recall from fine-grained candidate comparison.
  • 1.2B parameters comprise the primary configuration’s 0.6B encoder and 0.6B reranker.The paper uses standard encoder and reranker architectures rather than introducing a new architecture.
  • 37,979 synthetic query–skill pairs train the bi-encoder with all-field skill inputs and in-batch InfoNCE.Benchmark-labeled skills are excluded from training supervision, and the inventory is embedded offline for top-20 retrieval.
  • False-negative filtering removes functionally equivalent mined negatives using name deduplication, body overlap, and embedding similarity checks.The filtering addresses near-duplicate skills that would otherwise corrupt contrastive learning.
  • Listwise cross-entropy trains the reranker to compare 20 topical candidates against one another rather than score each independently.The listwise variant outperforms pointwise training by 30.7pp Hit@1.

5 Experiments

Experiments show that task-specific training makes compact, body-aware retrieval and reranking highly competitive on large skill-routing benchmarks, while routing improvements transfer to coding-agent task success and reduce serving cost.

  • Encoder results: 75.4% average R@20 gives SR-Emb-0.6B higher candidate coverage than Qwen3-Emb-8B at 72.6%.Higher coverage supplies more correct candidates to the reranking stage.
  • Main results: 74.0% average Hit@1 makes the 1.2B tuned pipeline exceed the 16B base pipeline’s 68.0% at 13× fewer parameters.The gain remains positive on both Easy (+8.0pp) and Hard (+4.0pp) tiers.
  • Main results: 74.0% Hit@1 for tuned reranking exceeds 71.4% with a base reranker and 68.0% for the 16B base pipeline.The 16B tuned scaling variant reaches 76.0% Hit@1, while the 1.2B system captures most of the gain.
  • Metric calibration: The strongest base pipeline remains better on strict multi-skill FC@10, .382 versus .353, so the main advantage is top-1 routing rather than exhaustive set recovery.Hit@1 improves on both single- and multi-skill queries.
  • Key ablations: Listwise reranking reaches 74.0% Hit@1 versus 43.3% for pointwise training, while false-negative filtering adds +4.0pp encoder Hit@1.These are the two reported training adaptations in the key ablations.
  • End-to-end transfer: Routing gains improve average task success across four coding agents in both top-1 and top-10 settings, with larger benefits for more capable agents.The end-to-end evaluation averages three trials across four agents.
  • Efficiency: 495.8 ms median latency makes the 1.2B pipeline 5.8× faster than the 16B base pipeline while using 15.8% less GPU memory.The serving benchmark used 80 timed queries from the real skill pool.

6 Related work

Related work covers large-scale tool retrieval, adaptive reranking, toolset synthesis, and skill organization, while this paper focuses on field visibility for natural-language skill bodies in large-pool routing.

  • Large-scale tool and skill retrieval: ToolRet evaluates 7.6K retrieval tasks over 43K heterogeneous tools, whereas this work isolates field visibility for natural-language skill bodies under progressive disclosure.The paper distinguishes skill-body routing from tool or API-document retrieval.
  • Toolset construction and reranking: CRAFT synthesizes specialized toolsets, and ToolRerank adaptively reranks tools using hierarchy and candidate difficulty.The paper’s architecture instead follows standard bi-encoder and cross-encoder neural IR.

7 Conclusion

At realistic registry scale, metadata-only routing misses critical body-resident signal, while a compact body-aware pipeline achieves strong routing and transfers gains to task execution. The study remains bounded by small-scale and limited downstream validation.

  • 37–44pp drops follow body removal across representative baselines, while body-distilled metadata and matched metadata-only fine-tuning recover only part of the deficit.
  • 74.0% Hit@1 is reached by a 1.2B body-aware retrieve-and-rerank pipeline at 5.8× lower latency than the strongest 16B base pipeline.
  • False-negative filtering and listwise loss are essential in homogeneous skill pools, according to the paper’s conclusion.
  • Routing gains transfer to direct task execution across four coding agents, with larger gains for more capable agents.
  • The 75-query core is small, downstream results cover four agents under one budget without significance testing, and transfer beyond the tested settings remains open.

A Evaluation details

The evaluation uses a 75-query expert-verified core drawn from SkillsBench and an approximately 80K-skill pool, with multi-skill metrics and robustness-oriented construction details. It also characterizes metadata richness and near-duplicate distractors.

  • Metrics: Hit@1 counts any ground-truth skill at rank 1, while Recall@K measures recovered ground-truth fraction and FC@10 requires the full set in the top 10.
  • Benchmark construction: 75 core queries remain after excluding 12 generic-only SkillsBench tasks, comprising 24 single-skill and 51 multi-skill queries.
  • Metadata richness: Descriptions are almost always present, but median descriptions contain 21 words versus 704 words for full bodies.
  • Robustness tiers: The Easy pool contains 78,361 candidate skills, and the benchmark also includes a Hard tier augmented with 780 LLM-generated distractors.
  • Pool construction: Same-name overlaps are intentionally retained after deduplication to preserve realistic near-duplicate confusion.

C Detailed attention analysis

The attention analysis controls for field length and examines whether the reranker’s body use reflects more than token quantity. Body attention is elevated for most queries, but the diagnostics do not establish token-level causality.

  • Figure 3: Figure 3 compares per-layer field attention trajectories with token-share baselines and checks query-level final-layer body attention against each query’s body-token baseline.
  • Figure 3: The short name field spikes above its 3.0% token-share baseline in middle layers, while final-layer attention returns to body.
  • Length-controlled attention: Final-layer body attention exceeds body token share on 69 of 75 queries and is effectively uncorrelated with absolute body length (r = 0.04).
  • Interpretation: The attention diagnostics argue against a trivial length effect but are not intended as a causal test of which tokens determine the final prediction.
  • Description-length stratification: The description-length analysis finds no monotonic gap decrease: the longest-description quartile still has a +31.8pp gap, while Q2 has the largest gap at +50.0pp.

E Oracle-ND construction and supplementary results

Oracle-ND replaces only ground-truth descriptions with body-grounded summaries to test whether metadata can recover hidden routing information. The broader pipeline uses body-aware retrieval and reranking with filtered negatives and listwise training.

  • Oracle-ND construction: Oracle-ND generates a query-blind 40–60-word functional description from each ground-truth skill’s name, description, and body, while leaving other pool entries unchanged.
  • Oracle-ND results: For the separately nd-fine-tuned encoder, Oracle-ND changes Hit@1 from 51.3% to 32.0%, a GT-only diagnostic not directly comparable to pipeline rows.
  • Supplementary results: On SkillBench-Supp, all-field routing retains a 5.1–12.1pp pipeline advantage and a 7.0–11.3pp encoder advantage over Oracle-ND.
  • Training adaptations: Training mines semantic, lexical, taxonomy, and random negatives, then removes false negatives using name, body-overlap, and embedding-similarity filters.
  • Reranking: The reranker uses listwise cross-entropy over 20 retrieved candidates because those candidates are often topically plausible.

H Top-K candidate ablation

The main benchmark selects K=20 candidates for reranking because it captures most candidate coverage while avoiding the weaker downstream Hit@1 observed at K=50. The reranker adds substantial top-1 recovery, but remaining errors include recall failures and multi-hop prerequisite inference.

  • H Top-K candidate ablation: K=20 captures most available candidate coverage and is used as the main benchmark operating point.The choice is supported by Figure 4 and Table 19 across the reported Easy/Hard comparisons.
  • H Top-K candidate ablation: −2.0pp average is the fine-tuned SR-Rank-0.6B drop when increasing from K=20 to K=50.Moving to K=50 does not improve downstream Hit@1 and often hurts it, while K=10 leaves less reranking headroom.
  • H Top-K candidate ablation: +8.7pp Hit@1 is the reranker’s net gain, increasing performance from 65.3% to 74.0%.Across 150 Easy+Hard evaluations, it fixes 19 cases where the encoder ranked the correct skill outside rank 1 but within top 20, while degrading 6 cases.
  • H Top-K candidate ablation: Cross-attention over implementation bodies rescues subtle matches, including environment setup and multi-format configuration conversion.The reranker promotes skills ranked 13 and 11 by the encoder by matching body-resident dependency or supported-format information.
  • H Top-K candidate ablation: Pointwise reranking can degrade a correct rank-1 retrieval to rank 18 in homogeneous candidate pools, whereas listwise scoring preserves rank 1.The pointwise model assigns similar scores to API-related candidates, effectively randomizing their order.
  • H Top-K candidate ablation: Multi-hop prerequisite inference remains a source of misses: all methods fail to connect invoice-fraud analysis with the required PDF table extraction skill.The case requires chaining fraud analysis, structured data needs, PDF invoices, and table extraction.

M.1 Representative downstream cases

Representative downstream cases show that routing affects agent execution through functional skill selection, while specialized multi-skill engineering tasks remain difficult for the compact router. The supplementary benchmark broadens evaluation across independently sourced skills and query-generation styles.

  • M.1 Representative downstream cases: 12/12 is SKILLROUTER’s top-1 success count versus 0/12 for the baseline on software-dependency-audit.The router selects the specific offline vulnerability-scanning workflow instead of a semantically similar general dependency-security skill.
  • M.1 Representative downstream cases: SKILLROUTER ranks speech-to-text first for video-tutorial-indexer, avoiding the baseline’s surface match to video-explorer.The task requires timestamped spoken-content extraction, so functional audio transcription is the relevant match.
  • M.1 Representative downstream cases: 8/12 versus 4/12 favors the baseline on the specialized HVAC-control case.The baseline recovers all five gold skills in top-10, while SKILLROUTER recovers only one; the case requires precise vocabulary across multiple engineering sub-disciplines.
  • M.1 Representative downstream cases: 256 supplementary queries are generated from three independent skill sources after automated leakage and length checks.The 100 ground-truth skills include 51 OpenClaw skills, 19 AgentSkillOS skills, and 30 pool-selected skills; the final queries have lengths from 43 to 226 words.
  • M.1 Representative downstream cases: Indirect queries are consistently harder across systems, with BM25 Hit@1 dropping from .083 to .034.This pattern supports the benchmark’s distinction between direct descriptive scenarios and queries requiring deeper capability inference.
Loading 2603.22455v5…