PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents

Ryangkyung Kang, Hongcheol Cho, Youngeun Kim

arXiv:2605.05726v3cs.AI

TL;DR

Skill retrieval from large skill libraries remains an underexplored challenge, especially under realistic inference constraints. SkillRet benchmarks retrieval across a curated skill library, and task-specific fine-tuning raises NDCG@10 to 86.44, a 12.9-point gain over the strongest prior retriever.

  • Problem

    Whether agents can identify relevant skills from large libraries under realistic inference constraints remains an underexplored research question.

    Source: §1 · p. 1

  • Method

    SkillRet provides a benchmark built from 16,129 public agent skills, with semantic tags, a two-level taxonomy, and training and evaluation samples.

    Source: §1 · p. 2

  • Results

    Task-specific fine-tuning on SkillRet reaches 86.44 NDCG@10, a 12.9-point gain over the strongest prior retriever.

    Source: §7 · p. 10

  • Takeaways & Limitations

    SkillRet supports controlled benchmarking and retrieval-oriented model development for skill retrieval.

    Source: §1 · p. 2

    Queries are synthetically generated rather than collected from live agent interactions, and may under-represent requests common in real deployments.

    Source: §6 · p. 10

Abstract

from arXiv · show

As LLM agents are increasingly deployed with large libraries of reusable skills, selecting the right skill for a user request has become a critical systems challenge. In small libraries, users may invoke skills explicitly by name, but this assumption breaks down as skill ecosystems grow under tight context and latency budgets. Despite its practical importance, skill retrieval remains underexplored, with limited benchmarks and little understanding of retrieval behavior on realistic skill libraries. To address this gap, we introduce SkillRet, a large-scale benchmark for skill retrieval in LLM agents. SkillRet contains 16,129 public agent skills, organized with structured semantic tags and a two-level taxonomy spanning 6 major categories and 18 sub-categories. It provides 63,259 training samples and 4,392 evaluation queries with disjoint skill pools, enabling both benchmarking and retrieval-oriented training. Across a diverse set of retrievers, we find that skill retrieval remains far from solved: off-the-shelf models struggle on realistic large-scale skill libraries, and prior skill-retrieval models still leave substantial headroom. Task-specific fine-tuning on SkillRet improves NDCG@10 by 12.9 points over the strongest prior retriever and by 16.2 points over the strongest off-the-shelf retriever. Our analysis further suggests that these gains arise because fine-tuned models better focus on the small skill-relevant signals within long and noisy queries. These results establish SkillRet as a strong benchmark and foundation for future research on retrieval in large-scale agent systems. We publicly release the benchmark (https://huggingface.co/datasets/ThakiCloud/SKILLRET), code (https://github.com/ThakiCloud/SKILLRET), and model checkpoints (0.6B: https://huggingface.co/ThakiCloud/SKILLRET-Embedding-0.6B; 8B: https://huggingface.co/ThakiCloud/SKILLRET-Embedding-8B).

1 Introduction

SkillRet frames skill retrieval as a distinct challenge for large-scale LLM agents and introduces a public benchmark for studying it. Its experiments show that task-specific fine-tuning improves retrieval, while analysis points to better focus on skill-relevant signals in noisy queries.

Motivation

  • Large skill libraries make explicit retrieval necessary because exposing the full library in context or expecting users to name the right skill is impractical.Retrieval selects a small relevant subset for each task, reducing context cost and supporting robust automated skill use at scale.Source: §1 · p. 1

Benchmark gap

  • Prior skill benchmarks [19] [7] [17] mainly assess end-to-end execution, while SkillRouter [39] has only 75 evaluation queries and does not publicly release its training data.SKILLRET isolates retrieval and provides larger-scale public training and evaluation splits than the closest skill-retrieval work.Source: §1 · p. 2, Table 1 · p. 2

Benchmark

  • SKILLRET contains 16,129 public skills, 63,259 training samples, and 4,392 evaluation samples, with tags and a taxonomy of 6 major and 18 sub-categories.The corpus was filtered from 22,795 listings and captures long-context skill documents and imbalanced skill distributions.Source: §1 · p. 2

Results

  • Task-specific fine-tuning yields substantial gains, with smaller fine-tuned models matching or surpassing much larger off-the-shelf models.Reranking helps most when the first-stage retriever has remaining headroom, with diminishing gains for stronger retrievers. Fine-tuned models focus better on skill-relevant sentences in long, noisy, compositional queries; on Terminal-Bench [21], retrieved skills preserve agent success while reducing cost by 10%.Source: §1 · p. 2

2 Related Work

Prior work establishes reusable skills and emerging skill ecosystems, while leaving skill retrieval difficult to isolate and compare as a standalone problem.

2.1 Agent Skills

  • Reusable skills are increasingly central to agent design, but ecosystem and usefulness benchmarks do not isolate retrieval quality.AgentSkillOS studies skill organization and orchestration [17], while SkillsBench finds curated skills improve performance but self-generated skills provide no average benefit [19]; SWE-Skills-Bench finds most public SWE skills provide little or no pass-rate improvement [7].Source: §2.1 · pp. 2–3

2.2 Skill Retrieval Benchmarks and Skill Routing

  • Skill retrieval work distinguishes tool retrieval, agent-facing pipelines, and routing models from SKILLRET’s standalone benchmark for comparing retrieval quality.ToolRet shows that strong conventional-IR models still struggle on tool retrieval [27], while SkillFlow proposes a multi-stage skill-retrieval pipeline [16]. SkillRouter studies routing over roughly 80K skills and finds that removing full skill bodies sharply reduces performance [39].Source: §2.2 · p. 3

3 SkillRet Benchmark

SkillRet constructs a large-scale benchmark of public agent skills by pairing a curated skill library with realistic natural-language queries and validated, disjoint training and evaluation pools.

3.1 Data Collection and Quality Filtering

  • Filtering reduces the initial 22,795-skill corpus to 17,810 curated skills through content eligibility checks and deduplication.Checks include description recovery, language and license filtering, content hashes, and name–description hashes; the exact-hash steps cannot detect differently worded functional equivalents.Source: §3.1 · pp. 3–4

3.2 Skill–Query Pair Generation

  • Queries are generated from randomly selected sets of one to three skills, with GAIA questions as style seeds and skill names omitted from generated requests.Inverse-frequency sampling first covers every skill before repeating any, and prior queries help suppress near-duplicates.Source: §3.2 · p. 4
  • Automatic leakage checks, multi-perspective LLM review, functional-equivalence auditing, and human validation filter query–skill pairs and prevent detected capability-level overlap across splits.Training queries use Qwen3.5-122B-A10B, while evaluation queries use Claude Opus 4.6 to reduce generator-style leakage into evaluation.Source: §3.2 · pp. 4–5

3 SkillRet Benchmark

  • The final benchmark contains 16,129 skills: 10,123 training and 6,006 evaluation skills, paired with 63,259 training queries and 4,392 evaluation queries.The splits have no exact skill overlap.Source: §3.1 · p. 4, Figure 1 · p. 5, §3.2 · p. 5

4 Benchmark Analysis

Section 4 describes the taxonomy and long-text skill corpus underlying retrieval, then characterizes how training and evaluation queries differ in length and number of required skills.

4.1 Taxonomy Overview

  • The five-stage taxonomy pipeline assigns the 17,810 curated skills to 6 major and 18 sub-categories.Experts validated a stratified sample with average accuracy of 95.5% for major categories and 92.2% for sub-categories; Software Engineering makes up 62.2% of the curated corpus.Source: §4.1 · pp. 5–6

4.2 Skill & Taxonomy Statistics

  • The retrieval target combines each skill’s name, description, and full Markdown, with median length 1,583 tokens (mean 2,083; 95th percentile 5,531; max 47,412).The corpus totals approximately 37.1 M tokens; Data & ML skills have the longest median length at 1,795 tokens, making retrieval a long-document matching problem.Source: §4.2 · p. 6

4.3 Query Statistics

  • Evaluation queries are longer than training queries (median 170 versus 72 words; 95th percentile 270 versus 108), and 54% require multiple skills.Training queries are sampled uniformly across k=1, 2, and 3, whereas evaluation queries comprise 46% single-skill, 40% two-skill, and 13% three-skill queries; their greater length likely challenges lexical matching.Source: §4.3 · pp. 6–7, Figure 2 · p. 6

5 Evaluation

SkillRet-Embedding substantially improves skill retrieval over prior and off-the-shelf retrievers. The evaluation also examines reranking, multi-skill queries, deployment costs, and how fine-tuning changes retrieval behavior.

5.1 Experimental Setup

  • The evaluation uses a retrieve-then-rerank pipeline with k=20 and encodes full skill documents, while fine-tuning embedding models on positive query–skill pairs and a reranker with mined hard negatives.It reports NDCG@10, Recall@10, and Completeness@10, and runs evaluations on one NVIDIA B200 GPU per model.Source: §5.1 · pp. 7–8

5.2 Experimental Results

  • SkillRet-Embedding-8B reaches 86.44 NDCG@10, leading SkillRouter-Embedding-0.6B [39] by 12.9 points and harrier-oss-v1-0.6b by 16.2 points.The functional-equivalence correction raises scores by a mean of 3.29 points across 19 models, with zero inversions among 171 model pairs.Source: §5.2 · p. 8
  • SkillRet-Reranker raises NDCG@10 from 86.44 to 87.74, while an oracle top-20 pool reaches 88.89, indicating limited remaining gains from candidate coverage.SkillRouter-Reranker [39] reaches 86.78 on the same candidate pool; its oracle gap is 1.27 points.Source: §5.2 · p. 8, Table 4 · p. 8

5.3 Analysis

  • SkillRet-Embedding-0.6B’s NDCG@10 advantage over Qwen3-Embedding-0.6B grows from 10.3 points at one required skill to 33.1 at three, with 42.1% three-skill completeness.Qwen3-Embedding-0.6B completes only 1.8% of three-skill queries.Source: §5.3 · p. 8, Table 5 · p. 8
  • Reranking is the latency bottleneck, while retrieval matches Terminal-Bench task completion within 0.3 percentage points and reduces mean cost per trial by 10%.Embedding takes 21.25 ms per query at 0.6B and 26.25 ms at 8B; reranking 20 candidates takes 831 ms at 0.6B and 1.88 s at 8B. The 8B embedder reaches 86.44 NDCG@10, and reranking adds 1.30 points.Source: §5.3 · p. 9, Table 6 · p. 9
  • Sentence-erasure results support the finding that fine-tuning concentrates retrieval signal on skill-relevant sentences: masking the single most important sentence causes a larger drop than the trained model’s 7.5-point full-query lead.The analysis masks sentences ranked by their effect on query-to-gold-skill similarity, then reruns retrieval.Source: §5.3 · p. 9, Table 7 · p. 9
  • MTEB Retrieval has a moderate correlation with SKILLRET NDCG@10 (Spearman ρ = 0.70), but frequent ranking inversions show that general retrieval rankings do not reliably predict skill retrieval performance.The analysis attributes the distinction to identifying capability signals in long, multi-sentence queries and reports task-specific fine-tuning as the most effective way to bridge the gap.Source: §5.3 · p. 9, Figure 3 · p. 10
  • SkillRet-Embedding-8B improves over Qwen3-Embedding baselines by 18.4–36.1 NDCG@10 points across major categories, yet Information Retrieval and AI Agents remain hardest, separated by 11.4 points from the easiest category.Sub-category results show within-major variance up to 11.5 points.Source: §5.3 · pp. 9–10, Table 8 · p. 10

6 Limitations

SKILLRET’s limitations are that its queries are synthetic and its evaluation isolates retrieval from downstream agent performance.

6 Limitations

  • Synthetic queries may under-represent terse, underspecified, conversational, or user-context-dependent requests common in real deployments.The benchmark uses GAIA-style seed examples, filters skill-name leakage, and validates query–skill pairs, but bridging synthetic benchmarks with live agent traffic remains future work.Source: §6 · p. 10
  • Retrieval quality is evaluated in isolation, without measuring downstream task success or end-to-end agent performance.The study leaves joint evaluation of skill retrieval and downstream execution to future work.Source: §6 · p. 10
  • Higher NDCG@10 does not necessarily imply better skill use, which also requires selecting, composing, interpreting, and executing retrieved skills under context and latency constraints.Source: §6 · p. 10

7 Conclusion … B.5 LLM-based Skill Assignment

SkillRet presents a large-scale benchmark for skill retrieval and reports substantial gains from domain-specific fine-tuning; its appendix documents the filtering, tagging, clustering, taxonomy design, and skill-assignment procedures used to build the corpus.

7 Conclusion

  • 86.44 NDCG@10 is achieved by SkillRet domain-specific fine-tuning, versus 73.54 for the strongest prior skill-retrieval model and 70.26 for the strongest off-the-shelf retriever.These correspond to gains of +12.9 and +16.2 points, respectively; the benchmark targets long-form skill documents matched against small actionable signals in user queries.Source: §7 · p. 10

A Data, code, and model.

  • The dataset, code, and 0.6B and 8B model weights are publicly released for retrieval evaluation and model development.The dataset is derived from public GitHub-hosted skills and synthetic queries, and is not intended to profile authors, infer personal attributes, or certify end-to-end agent safety.Source: A Data, code, and model · p. 14

B.1 Per-Step Filtering Attrition

  • Content deduplication and language filtering produce the largest reductions in the five-stage quality-filtering pipeline.The appendix describes this filtering as part of the broader construction process covering filtering, tagging, clustering, taxonomy design, assignment, and human validation.Source: §B.1 · p. 14, Table 9 · p. 14, B Dataset Construction Details · p. 14

B.2.1 Pass 1: Category Discovery

  • The first tagging pass analyzes 17,810 skill names with truncated descriptions and discovers 13 action, 14 object, and 13 domain categories for three structured tag dimensions.The categories are corpus-discovered rather than predefined, with the target granularity specified in the prompt and label sets manually reviewed for coherence and ambiguity.Source: §B.2.1 · p. 14, §B.2 · p. 14

B.2.2 Pass 2: Batch Classification

  • In the second pass, Claude Sonnet 4.6 classifies skills in batches of 100 using the discovered labels, yielding 17,810 action-object-domain tuples after deduplication.The procedure assigns one label per dimension, operationalizing the three structured tags introduced for each skill.Source: §B.2 · p. 14, §B.2.2 · p. 15

B.3 Consensus Clustering over Action–Object Combinations

  • Consensus clustering links action-object combinations that co-occur across all five clustering resolutions, producing 10 stable groups and 19 singletons.The top 44 of 182 combinations cover 80% of skills; the stable groups include object-bound and action-bound patterns and seed the initial taxonomy draft.Source: §B.3 · p. 17

B.4 Iterative Taxonomy Design

  • Expert review of stratified samples refined the initial 7-major, 21-subcategory taxonomy into the final 6-major, 18-subcategory structure.Review rounds addressed structural ambiguities, including an over-broad Documentation & Knowledge category and scattered ML-related skills.Source: §B.4 · p. 17

B.5 LLM-based Skill Assignment

  • Because action, object, and domain tags could misroute skills whose purpose depends on their descriptions, Claude Sonnet 4.6 assigned all 17,810 skills to taxonomy pairs using names and descriptions.The assignment principle is to classify by the domain where a capability is used, rather than by the fact that the capability extends an agent.Source: §B.5 · pp. 17–18

B.6 Human Validation of Assignment … C.2 Multi-Perspective LLM Review

The paper validates LLM-assigned taxonomy labels with expert review, then describes how it generates realistic skill-dependent queries and screens them through multiple LLM review perspectives.

B.6 Human Validation of Assignment

  • Experts independently assessed whether assigned major and sub-category labels were appropriate on a stratified 200-skill sample covering all six major categories.Source: §B.6 · p. 19
  • Reviewers averaged 95.5% accuracy for major categories and 92.2% for sub-categories, with full agreement on 91.0% and 84.5% of items, respectively.Experts independently judged a stratified sample of 200 skills, preserving the distribution across all six major categories.Source: §B.6 · p. 19

C.1 Query Generation Prompt

  • Generation calls include sampled skill names and descriptions, a random subset of 165 GAIA [23] validation questions, and up to 30 prior queries for that skill to reduce near-duplicates.There is no system prompt; each instruction is sent in a single user turn, and unrealistic skill combinations are discarded by returning None.Source: §C.1 · p. 20
  • The prompt requests a standalone message requiring all listed skills, varies user-query styles, and forbids naming skills or repeating prior queries.It also allows the model to output None when a combination cannot fit a realistic scenario.Source: §C.1 · p. 20

C.2 Multi-Perspective LLM Review

  • Three separate API-call prompts evaluate skill coherence, query quality, and benchmark discriminability as independent review dimensions.The workflow uses Claude Sonnet 4.6 for each query–skill pair that passes the leakage filter.Source: §C.2 · p. 20
  • The coherence review checks whether the query semantically matches and genuinely requires every listed skill, marking unrelated or mismatched combinations invalid.Reviewers are instructed to be conservative and mark invalid only when clearly problematic.Source: §C.2 · p. 21
  • The query-quality review checks realism, specificity, answerability, professional plausibility, and technical coherence.It flags queries that are too vague or unrealistic.Source: §C.2 · p. 21
  • The discriminability review rejects generic queries that a capable model could answer adequately without the specific skill.A query is invalidated when at least two reviewers return invalid; one invalid verdict instead sends it to human expert review.Source: §C.2 · pp. 20–21

D Functional-Equivalence and Graded-Relevance Audit

The audit finds that removing functionally substitutable evaluation skills and training leakage raises measured NDCG@10, while treating evaluation substitutes as graded-relevance labels changes scores very little. Functional twins are rare in training batches, and complementary checks probe whether gains reflect retrieval quality rather than benchmark construction.

D Functional-Equivalence and Graded-Relevance Audit

  • Correcting evaluation equivalences and train-pool leakage raises NDCG@10 by 3.29 points on average across 19 first-stage systems.The average increase has a standard deviation of 0.33 and ranges from 2.70 to 3.91 points.Source: D Functional-Equivalence and… · p. 22
  • Only 6.7% of queries contain substitute labels, and graded-minus-binary NDCG@10 stays within 0.09 points across 19 systems (mean −0.01).The graded variant retains evaluation-internal functional equivalents at relevance level 1, the seed skill at level 2, and unrelated skills at level 0.Source: Table 12 · p. 22, D Functional-Equivalence and… · p. 22
  • Strictly judged functional twins become in-batch negatives in approximately 0.3% and 0.06% of instances at effective batch sizes of 384 and 80, respectively.A conservative bound counting partially overlapping pairs raises these rates to below 1.7% and 0.4%; corrected evaluation, cross-benchmark transfer, and unlabeled Terminal-Bench offer complementary checks on retrieval quality.Source: D Functional-Equivalence and… · p. 22

E Top-k Reranking Depth Ablation

Increasing reranking depth improves NDCG@10 across the tested first-stage retrievers and rerankers, while k=20 is used in the main experiments as a practical performance-cost trade-off.

E Top-k Reranking Depth Ablation

F Document Representation Ablation … M Qualitative Visualization of Sentence Erasure Importance

These appendices show that full-document encoding improves retrieval, describe the models’ sequence-length and prompt choices and SkillRet fine-tuning, and examine how performance relates to general retrieval and varies by sub-category. They also outline a sentence-erasure analysis for identifying query sentences that contribute to retrieving a gold skill.

F Document Representation Ablation

G Model Maximum Sequence Lengths

  • Encoder-only models’ 512-token limit truncates most skill documents, whereas decoder-only models’ 8K–32K contexts cover nearly all documents.Maximum sequence lengths follow official model cards or documentation, or the model’s default context window when no explicit limit is stated.Source: G Model Maximum Sequence Lengths · p. 23, Table 15 · p. 23

H Retrieval Prompts for Each Evaluated Model

  • Most models use recommended query/document prompts, while Harrier-OSS, Qwen3-Embedding, and Qwen3-Reranker use skill-retrieval instructions instead of default web-search instructions.Octen-Embedding-8B also prepends “- ” to each document to work around an upstream tokenizer issue.Source: H Retrieval Prompts for Each Evaluated… · p. 24, Table 16 · p. 24

I Fine-tuning Base Model Selection

  • Fine-tuned Qwen3-Embedding and harrier-oss-v1-0.6b variants perform nearly identically across metrics, so Qwen3-Embedding is selected as the base to avoid double fine-tuning and preserve cleaner provenance.The same rationale leads to choosing Qwen3-Embedding-8B over Qwen3-based Octen-Embedding-8B at the 8B scale.Source: I Fine-tuning Base Model Selection · p. 24, Table 17 · p. 25

J SkillRet Fine-tuning Details

  • SkillRet embedding models are fine-tuned for one epoch on 63,259 synthetic queries and 127,190 positive query–skill pairs, with training and evaluation skills disjoint.The embedding models use MultipleNegativesRankingLoss and the evaluation document representation and query instruction. The reranker uses binary cross-entropy on yes/no token probabilities and hard negatives mined from SkillRet-Embedding-0.6B.Source: J SkillRet Fine-tuning Details · p. 25

K MTEB Retrieval vs. SKILLRET Performance

  • Table 18 compares models’ MTEB Retrieval scores with SKILLRET NDCG@10, ordering the models by descending MTEB Retrieval score.Source: Table 18 · p. 26

L Per-Sub-category Retrieval Performance

  • Table 19 breaks down NDCG@10 and Recall@10 across all 18 sub-categories for base and fine-tuned Qwen3-Embedding models.Sub-categories are grouped by Major category and sorted by fine-tuned 8B NDCG@10 within each group; queries count in every sub-category represented among their gold skills.Source: L Per-Sub-category Retrieval Performance · p. 25, Table 19 · p. 26

M Qualitative Visualization of Sentence Erasure Importance

  • Sentence-erasure analysis measures each query sentence’s contribution to retrieving the gold skill by the similarity drop after replacing it with [MASK].A larger drop indicates a stronger contribution.Source: M Qualitative Visualization of… · p. 26

N Multi-Skill Retrieval by Category · O Retrieval Latency and Throughput · P Broader Impacts

The section reports how multi-skill fine-tuning gains vary by category, specifies latency and throughput measurement conditions, and frames SkillRet as retrieval research support rather than evidence of deployed-agent safety.

N Multi-Skill Retrieval by Category

  • For an example query, sentence-erasure importance concentrates more on the skill-relevant sentence in the trained model, while the base model distributes importance more diffusely.The figure measures importance as the similarity drop after replacing a sentence with [MASK].Source: Figure 6 · p. 27
  • Information Retrieval's NDCG@10 base scores are 44.5, 43.5, and 44.6, so its gain stays near the k=1 value of 32.3 points.The category's skills are the corpus's shortest and its smallest major category; the passage attributes its difficulty to interchangeable documents rather than the number of required skills.Source: N Multi-Skill Retrieval by Category · p. 27
  • In five of six major categories, the NDCG@10 fine-tuning gain widens monotonically from single-skill to three-skill queries; Data & ML's gain widens by 29.2 points.Data & ML's base score falls 41.6 points from k=1 to k=3.Source: N Multi-Skill Retrieval by Category · pp. 26–27, Table 20 · p. 27

O Retrieval Latency and Throughput

  • Fine-tuned and SkillRouter checkpoints share the runtime of their corresponding Qwen architectures because fine-tuning does not change the architecture.Source: O Retrieval Latency and Throughput · p. 27
  • Embedding latency uses batch size 1; throughput uses 1,024 queries at each model's largest fitting batch size, making throughput comparable within—but not across—model sizes.Measurements use 100 fixed queries after 10 warm-up iterations on one NVIDIA B200 with HuggingFace Transformers, FlashAttention-2, and bfloat16.Source: O Retrieval Latency and Throughput · p. 27, Table 21 · p. 28
  • Reranking latency covers one query with 20 candidates; models fitting batch size 20 score them in one pass, while Qwen3-Reranker-8B requires five passes.Its five-pass latency is included in the reported value; BM25 runs on CPU as a reference.Source: O Retrieval Latency and Throughput · p. 27, Table 22 · p. 28

P Broader Impacts

  • SkillRet isolates retrieval quality to support research on selecting procedural knowledge, but retrieval performance does not establish safe or reliable end-to-end agent behavior.Its public GitHub-hosted skills and synthetic queries are for retrieval evaluation and model development, not author profiling or downstream safety certification; deployments should add provenance checks, permission controls, sandboxing, and human oversight for high-impact actions.Source: P Broader Impacts · p. 28

Related papers

Loading 2605.05726v3…