PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents

Ryangkyung Kang, Hongcheol Cho, Youngeun Kim

arXiv:2605.05726v3cs.AI

TL;DR

Large libraries of reusable agent skills make selecting the right skill a retrieval problem that existing benchmarks and models do not adequately address. SKILLRET builds a large-scale benchmark with filtered public skills, generated and validated queries, and controlled splits; fine-tuning substantially improves retrieval performance.

  • Problem

    Large skill libraries make explicit invocation impractical, motivating retrieval of a small relevant subset under context and latency constraints.

  • Method

    SKILLRET constructs a benchmark from public skills using filtering, LLM-generated realistic queries, audits, and validation, with disjoint training and evaluation pools.

  • Results

    Skill-specific fine-tuning reaches 86.44 NDCG@10, improving by 12.9 points over the strongest prior retriever and 16.2 points over the strongest off-the-shelf retriever.

  • Takeaways & Limitations

    SKILLRET establishes skill retrieval as a distinct retrieval problem and a foundation for research in large-scale agent systems.

  • Takeaways & Limitations

    SKILLRET queries are synthetically generated and may under-represent terse, underspecified, conversational, or user-context-dependent requests from live deployments.

Abstract

from arXiv · show

As LLM agents are increasingly deployed with large libraries of reusable skills, selecting the right skill for a user request has become a critical systems challenge. In small libraries, users may invoke skills explicitly by name, but this assumption breaks down as skill ecosystems grow under tight context and latency budgets. Despite its practical importance, skill retrieval remains underexplored, with limited benchmarks and little understanding of retrieval behavior on realistic skill libraries. To address this gap, we introduce SkillRet, a large-scale benchmark for skill retrieval in LLM agents. SkillRet contains 16,129 public agent skills, organized with structured semantic tags and a two-level taxonomy spanning 6 major categories and 18 sub-categories. It provides 63,259 training samples and 4,392 evaluation queries with disjoint skill pools, enabling both benchmarking and retrieval-oriented training. Across a diverse set of retrievers, we find that skill retrieval remains far from solved: off-the-shelf models struggle on realistic large-scale skill libraries, and prior skill-retrieval models still leave substantial headroom. Task-specific fine-tuning on SkillRet improves NDCG@10 by 12.9 points over the strongest prior retriever and by 16.2 points over the strongest off-the-shelf retriever. Our analysis further suggests that these gains arise because fine-tuned models better focus on the small skill-relevant signals within long and noisy queries. These results establish SkillRet as a strong benchmark and foundation for future research on retrieval in large-scale agent systems. We publicly release the benchmark (https://huggingface.co/datasets/ThakiCloud/SKILLRET), code (https://github.com/ThakiCloud/SKILLRET), and model checkpoints (0.6B: https://huggingface.co/ThakiCloud/SKILLRET-Embedding-0.6B; 8B: https://huggingface.co/ThakiCloud/SKILLRET-Embedding-8B).

1 Introduction

Large skill libraries make explicit skill invocation impractical, creating a retrieval problem that existing benchmarks and evaluations address only weakly. SKILLRET responds with a large, public benchmark and shows substantial gains from task-specific fine-tuning.

  • Motivation: Large default skill pools make exposing every skill in context or expecting users to know the correct skill impractical.Retrieval selects a small relevant subset while reducing context cost and supporting automated skill use.
  • Problem: Skill retrieval is a central systems problem: agents must identify appropriate skills from large libraries under realistic inference constraints.
  • Problem: Existing benchmarks mainly evaluate end-to-end execution, while retrieval benchmarks target tools or remain small; SkillRouter has only 75 evaluation queries and unreleased training data.
  • Contribution: SKILLRET contains 16,129 public agent skills, 63,259 training samples, 4,392 evaluation samples, and a taxonomy spanning 6 major and 18 sub-categories.The benchmark supports controlled benchmarking and retrieval-oriented model development.

2 Related Work

Related work increasingly treats reusable skills as core components of agent systems, while retrieval-focused studies establish retrieval as an agent bottleneck. SKILLRET differs by isolating large-scale skill retrieval with public, controlled evaluation.

  • Reusable skill ecosystems: MetaClaw, XSkill, and WebXSkill develop reusable skills or knowledge for continual, multimodal, and web-agent settings.These systems emphasize inference-time access to skill libraries or reusable knowledge.
  • Reusable skill ecosystems: AgentSkillOS studies ecosystem-scale organization, selection, and orchestration, while SkillsBench evaluates whether skills improve performance across tasks and domains.
  • Retrieval benchmarks: ToolRet shows that strong conventional information-retrieval models can struggle with tool retrieval, motivating retrieval as a first-class agent bottleneck.
  • Retrieval benchmarks: Unlike ToolRet, SKILLRET targets long-form procedural skills with reusable prompting logic and compositional structure.
  • Retrieval benchmarks: Unlike SkillFlow and SkillRouter, SKILLRET isolates retrieval as a standalone benchmark with public train/evaluation splits and controlled ranking-based evaluation.SkillRouter uses roughly 80K candidates but reports only 75 expert-verified queries.

3 SkillRet Benchmark

SKILLRET constructs a large skill-retrieval benchmark by filtering public skills, generating realistic multi-skill queries, and auditing query–skill pairs. Its disjoint splits support controlled training and evaluation at scale.

  • Data collection and filtering: The benchmark starts from 22,795 community-contributed skills and applies quality filtering and deduplication before query generation.The filtering pipeline retains 17,810 curated skills, or 78.1% of the raw corpus.
  • Validation: Automatic checks, LLM review, functional-equivalence auditing, capability-leakage auditing, and human validation filter generated query–skill pairs.Functional-equivalence auditing groups substitutable evaluation skills and retains one representative per class while preserving valid answers.
  • Splits: The final benchmark contains 16,129 skills, with 10,123 training skills and 6,006 evaluation skills paired with 4,392 queries.
  • Skill–query pair generation: Query generation uses a self-instruct-style LLM pipeline prompted to create realistic natural-language requests requiring one or more library skills.GAIA validation questions provide 165 style seeds, and skills are sampled with inverse-frequency weighting.
  • Splits: Training and evaluation use disjoint skill pools, while split category distributions deviate by less than 1 percentage point from the full library.

4 Benchmark Analysis

SKILLRET combines structured taxonomy construction with detailed skill and query statistics to characterize large-scale retrieval. The benchmark uses long skill documents, varied query lengths, and an evaluation set dominated by multi-skill requests.

  • Taxonomy construction: 17,810 curated skills receive three structured tags, clustering-based taxonomy construction, LLM assignment, and human validation.The resulting taxonomy spans 6 major categories and 18 sub-categories.
  • Skill statistics: 1,583 tokens is the median composite skill length, with a 5,531-token 95th percentile and 47,412-token maximum.The composite includes the skill name, description, and full Markdown body; Data & ML skills are longest by median length.
  • Query statistics: 170 versus 72 words is the median evaluation-versus-training query length, while the corresponding 95th percentiles are 270 versus 108 words.The evaluation queries were generated by Claude Opus 4.6 and the training queries by Qwen3.5-122B-A10B.
  • Evaluation design: The corrected evaluation set contains 6,006 skills and 4,392 queries after remapping functional equivalents and removing train–evaluation functional overlaps.Table 3 reports first-stage retrieval results on this corrected set.
  • Query statistics: 54% of pre-audit evaluation queries require multiple skills, with 46% single-skill, 40% two-skill, and 13% three-skill queries.Training queries are sampled uniformly across k ∈{1, 2, 3}, whereas evaluation queries concentrate on lower k values.

5 Evaluation

SKILLRET evaluates skill retrieval with a two-stage pipeline across diverse models, showing that domain-specific fine-tuning substantially improves retrieval while reranking and latency create deployment trade-offs. Analyses further examine multi-skill robustness, query-signal concentration, category variation, and correspondence with MTEB Retrieval.

  • Experimental Setup: The evaluation uses retrieve-then-rerank with cosine-similarity retrieval, k=20 candidates, and full skill-document encoding under model-specific sequence limits.Larger k improves coverage but increases reranking cost; full document text outperforms name-and-description-only encoding.
  • Embedding Retrieval: 86.44 NDCG@10 from SkillRet-Embedding-8B exceeds SkillRouter-Embedding-0.6B by 12.9 points and harrier-oss-v1-0.6b by 16.2 points.SkillRet-Embedding-0.6B reaches 81.12 NDCG@10, while the strongest off-the-shelf model reaches 70.26.
  • Multi-skill Queries: 33.1 points is the SkillRet-Embedding-0.6B NDCG@10 advantage at three required skills, versus 10.3 points for single-skill queries.Three-skill completeness reaches 42.1%, while off-the-shelf Qwen3-Embedding-0.6B completes only 1.8% of three-skill queries.
  • Deployment Efficiency: 21.25 ms is the 0.6B embedder’s per-query latency, compared with 831 ms for reranking 20 candidates.The 8B embedder requires 26.25 ms and reaches 86.44 NDCG@10; reranking adds only 1.30 points, making it the principal bottleneck.
  • Query Analysis: The trained model’s retrieval signal concentrates on a small set of skill-relevant sentences rather than being distributed across the whole query.Removing the single most important sentence causes a larger performance drop than the trained model’s 7.5-point full-query advantage over the base model.
  • Cross-benchmark Analysis: MTEB Retrieval score has a moderate positive correlation with SKILLRET NDCG@10, but frequent ranking inversions limit its predictive value for skill retrieval.Figure 3 compares the two scores, with circle size proportional to parameter count; the reported Spearman correlation is ρ = 0.70.
  • Category Analysis: Taxonomy stratification reveals category disparities and sub-category variation of up to 11.5 points that aggregate NDCG@10 can hide.The reported aggregate NDCG@10 is 86.4, including for SkillRet-Embedding-8B.

6 Limitations

SKILLRET’s limitations concern the realism of its synthetic queries and its isolation of retrieval quality from downstream agent execution.

  • Query Realism: SKILLRET uses synthetically generated queries, which may under-represent terse, underspecified, conversational, or user-context-dependent requests from live deployments.The authors identify bridging synthetic benchmarks with real agent traffic as future work.
  • Evaluation Scope: SKILLRET evaluates retrieval quality in isolation and does not establish that higher NDCG@10 yields better skill use or downstream task success.Retrieved skills must still be selected, composed, interpreted, and executed under practical context and latency constraints.

7 Conclusion

SKILLRET establishes a large-scale benchmark for skill retrieval, combining a curated skill corpus, structured taxonomy, and retrieval-oriented data. Its construction uses filtering, LLM-based tagging, iterative taxonomy refinement, and human-in-the-loop validation.

  • Conclusion: SKILLRET contains 16,129 curated skills, 6 Major categories, 18 Sub-categories, 63,259 training queries, and 4,392 evaluation queries.The benchmark is derived from 22,795 initial listings and includes a corrected evaluation set of 6,006 skills.
  • Validation and Release: Human validation and public release support the benchmark’s use for retrieval evaluation and model development.The dataset, code, and pretrained model weights are publicly released.
  • Corpus Construction: The filtering pipeline’s largest reductions occur during language filtering and content deduplication.These reductions are reported as Steps 2 and 4 of the five-stage quality-filtering pipeline.
  • Taxonomy Construction: The skill library is organized with primary_action, primary_object, and domain tags assigned through LLM-based classification using discovered closed label sets.The procedure first discovers categories from the corpus, then classifies skills in batches and removes double-tagged duplicates.
  • Taxonomy Construction: The final taxonomy is refined through iterative expert review of stratified skill samples, resolving ambiguities across categories and classification axes.Consensus clusters seed an initial taxonomy, which experts revise into 6 Major and 18 Sub-categories.

B.6 Human Validation of Assignment

Human experts validate the quality of LLM-based taxonomy assignment on a stratified sample of 200 skills. The evaluation uses independent judgments of Major and Sub-category appropriateness, alongside multi-person review of generated benchmark queries.

  • Assignment Validation: Experts independently judge whether each skill’s assigned Major and Sub-category are appropriate in a stratified sample of 200 skills.The sample preserves the corpus-level distribution across all six Major categories.
  • Assignment Validation: 95.5% average accuracy is achieved for Major categories and 92.2% for Sub-categories.Full three-way agreement reaches 91.0% for Major categories and 84.5% for Sub-categories.
  • Query Validation: Generated query–skill pairs are reviewed through three independent LLM-based evaluations covering Skill Coherence, Query Quality, and Benchmark Discriminability.Queries receiving at least two invalid verdicts are rejected, while a single invalid verdict routes the query to human expert review.
  • Query Validation: The query review checks realism, specificity, technical coherence, skill alignment, and whether specialized skill access is needed.The reviewers assess both whether queries require the listed skills and whether they can distinguish models with and without those skills.

D Functional-Equivalence and Graded-Relevance Audit

The audit addresses whether functional-equivalent skills distort evaluation or training and complements corrected evaluation with cross-benchmark and label-free checks. It also documents the graded-relevance evaluation variant.

  • Functional-Equivalence Audit: Functional-equivalent evaluation skills are identified by retrieving nearest neighbors and using GPT-5.5 to judge functional substitutability.Union–find forms equivalence groups within the evaluation pool, retaining one representative per group.
  • Graded-Relevance Audit: Table 12 reports binary and graded NDCG@10 on the graded-relevance variant.
  • Functional-Equivalence Audit: The multi-positive training set contains 127,190 positive query-skill pairs across 63,259 queries, or approximately two skills per query.A functional twin becomes an in-batch negative only when it appears in the same batch.
  • Functional-Equivalence Audit: Functional twins appear as in-batch negatives in approximately 0.3% and 0.06% of instances for effective batch sizes of 384 and 80.Under a conservative bound including partially overlapping pairs, the rates remain below 1.7% and 0.4%.
  • Audit Checks: The corrected evaluation, cross-benchmark transfer, and Terminal-Bench provide complementary checks that gains reflect retrieval quality rather than benchmark construction conventions.Terminal-Bench carries no relevance labels.

E Top-k Reranking Depth Ablation

The reranking-depth ablation evaluates three candidate depths across first-stage retrievers and two rerankers. Larger candidate pools consistently improve NDCG@10, while k=20 is selected as the practical operating point.

  • Reranking Depth: Larger k consistently improves NDCG@10 across all three first-stage retrievers and both rerankers.The ablation evaluates k ∈{10, 20, 50} with Qwen3-Reranker-0.6B and Qwen3-Reranker-8B.
  • Reranking Depth: k=20 is adopted in the main experiments as a practical trade-off between retrieval performance and computational cost.

F Document Representation Ablation

Full-text skill representations outperform name-and-description representations, while context limits and prompt choices shape how models process long skill documents. The evaluation also relates SkillRet performance to general retrieval ability and examines category-level behavior.

  • Document Representation: 1.5–11.4 NDCG@10 points separate Full-text encoding from Name+Desc encoding across all three embedding models.Full encoding includes the skill name, description, and Markdown body up to each model’s maximum sequence length.
  • Prompting: Models generally use prompts from their official documentation, with three model families receiving skill-retrieval-specific prompt or formatting changes.Harrier-OSS, Qwen3-Embedding, and Qwen3-Reranker replace web-search instructions; Octen-Embedding-8B receives a tokenizer workaround.
  • Fine-Tuning Setup: Fine-tuning Qwen3-Embedding and harrier-oss-v1-0.6b produces nearly identical performance at 0.6B, so Qwen3-Embedding is selected for cleaner experimental provenance.The same rationale is applied at the 8B scale.
  • Training: SkillRet training uses 63,259 synthetic queries, 127,190 positive query–skill pairs, and disjoint training and evaluation skills.Embedding models are fine-tuned with MultipleNegativesRankingLoss, pairing each query with positive skill documents.
  • Generalization: SkillRet NDCG@10 only moderately tracks MTEB Retrieval, with notable exceptions showing that skill retrieval differs from general information retrieval.harrier-oss-v1-0.6b achieves the best off-the-shelf SkillRet score despite ranking fourth on MTEB Retrieval.

N Multi-Skill Retrieval by Category

Fine-tuning gains generally widen as queries require more skills, especially in categories where base retrieval degrades sharply. Information Retrieval is an exception because its difficulty reflects interchangeable documents rather than query skill count, while deployment implications require caution.

  • Multi-Skill Retrieval: In five of six major categories, fine-tuning gains widen monotonically from single-skill to three-skill queries.Categories with the largest base-score declines show the largest growth in fine-tuning gains.
  • Mechanistic Analysis: Sentence erasure analysis measures each sentence’s contribution through the similarity drop after replacement with [MASK].The qualitative example shows the trained model concentrating importance more strongly on the skill-relevant sentence than the base model.
  • Multi-Skill Retrieval: Data & ML falls 41.6 points from k=1 to k=3, while its fine-tuning gain widens by 29.2 points.Here, increasing the number of required skills is accompanied by both declining base performance and expanding fine-tuning benefit.
  • Information Retrieval: Information Retrieval remains near its k=1 fine-tuning gain of 32.3 points because its base scores stay around 44.5, 43.5, and 44.6.The passage attributes this difficulty to short, interchangeable documents rather than the number of skills required.
  • Efficiency: Embedding latency is measured at batch size 1, while throughput uses the largest batch size each model fits on the same GPU.Reranking latency measures one query’s top-20 documents; comparisons are most directly comparable within the stated measurement setup.
  • Broader Impacts: Strong retrieval performance does not establish safe or correct end-to-end agent behavior.The benchmark cautions that retrieved skills may be outdated, unsafe, misapplied, or incorrectly composed, requiring safeguards such as provenance checks and human oversight.
Loading 2605.05726v3…