SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale

• Public skill aggregations reached 238,180 skills, while adding skills to a curated bundle produced non-monotonic pass-rate changes, motivating precise selection.The reported lift was +17.8% with one relevant skill, +18.6% with two-to-three, and +5.9% with four or more.Source: §1 · p. 1
• Liu et al. (2026b) use BM25 and dense retrieval with agent-side query rewriting, candidate exploration, and skill refinement inside the task loop.Their loop can compose a refined skill from partially relevant skills when no single pool skill suffices, a synthesis static retrieval cannot produce.Source: §1 · pp. 1–2
• On three of four settings, plain BM25 matched or exceeded Liu et al. (2026b)’s refined loop; SkillSeek covered the fourth, with spend of $27.54 versus $51.30.On the 34K / Qwen3.5 setting, a Qwen3-Reranker-0.6B variant matched liu_refined at 0.442 pass rate; the default bge-reranker-v2-m3 variant recorded +5.7% over no skills versus liu_refined’s +9.0%.Source: §1 · p. 2
• SkillSeek applies a standard two-stage IR recipe: a BGE-base bi-encoder feeds a small cross-encoder, exposed over MCP.The authors characterize the contribution as empirical and open-source.Source: §1 · p. 2

Agent Skills

• SkillsBench reports that curated skill bundles add +16.2% on average across seven model-harness configurations, while bundles of four or more regress to +5.9%.Han et al. (2026) found that 39 of 49 public SWE skills yielded zero pass-rate change, while seven produced meaningful gains on domain-specific tasks.Source: §2 · p. 3

Skill Selection and Retrieval

• Liu et al. (2026b)’s marketplace-scale retrieval recovered most of the curated-to-34K pool loss through query-specific refinement, but used an LLM-mediated exploration pass per task.For Claude Opus 4.6, pass rate changed from 51.2% with curated skills to 40.1% on the 34K pool, then to 48.2% with refinement.Source: §2 · p. 3
• Zheng et al. (2026) report 74.0% Hit@1 with a body-aware retrieve-and-rerank pipeline on an approximately 80K pool; their routing-accuracy metric differs from agent pass rate.They fine-tune their encoder and reranker and report neither an LLM-mediated retrieval comparison nor token or monetary cost; the paper notes routing accuracy and agent pass rate can move in opposite directions.Source: §2 · p. 3

Creating and Securing the Skill Pool

• Other work addresses pool authoring and trust, including skill compression, bundle optimization, automated mining, injection attacks, abandoned repositories, and pre-load risk scoring.Reported examples include 71.1% novelty for automatically mined skills and 0.800 F1 for pre-load risk scoring.Source: §2 · p. 3

3 SkillSeek

• SkillSeek is exposed as an MCP server so MCP-supporting harnesses can adopt it without source-level changes.The paper names Claude Code, Codex CLI, and OpenHands as examples.Source: §3 · p. 3

3.1 Two-Stage Retriever

• The two-stage retriever embeds the query and indexed skills, selects the top 20 by cosine similarity, then reranks those candidates and returns the top five.Stage one uses a 110M-parameter BGE-base bi-encoder; stage two uses a 568M-parameter bge-reranker-v2-m3 cross-encoder.Source: §3.1 · p. 3
• SkillSeek indexes skill names and descriptions with a four-tag Tool-REX v3 profile, while fetching the SKILL.md body only on demand.The profile adds file-type and operation surface forms; the paper states that appending the raw body adds noise.Source: §3.1 · pp. 3–4

3.2 MCP Server Interface

• The MCP interface provides skill_lookup, skill_load, and skill_list; a server-side method header lets the experimental driver change retrieval backends without changing the agent’s tool schema.Source: §3.2 · p. 4

3.3 Driver and Harness

• Experiments use the OpenHands SDK directly, whose in-process loop enables per-turn event capture and native MCP support enables backend swaps between conditions.Source: §3.3 · p. 4

4.1 Experimental Setup

• The evaluation covers 89 SkillsBench tasks in a 192-skill curated pool and a 34K marketplace pool, using OpenHands and four retrieval-method families.The curated pool measures retrieval when relevant skills are in the catalog; the marketplace pool tests finding them among many irrelevant skills. Each trial is a single agent run with a deterministic, graded verifier.Source: §4.1 · pp. 4–5

4.2 Main Results

• Lightweight methods lead on the curated 192-skill pool, while heavier rerankers and LLM refinement catch up at marketplace scale.The same pool-size split appears between liu_hybrid and liu_refined: hybrid leads on the 192-skill pool, while refinement narrowly leads on 34K/Qwen3.5.Source: §4.2 · p. 5
• Across the four pool/backbone settings, bm25 matches or exceeds liu_refined on three; on 34K/Qwen3.5 both SkillSeek’s Qwen3-Reranker-0.6B and liu_refined score 0.442.These are observed single-trial differences, not resolved advantages; Table 1 reports mean pass rate over 89 tasks per setting.Source: §4.2 · p. 5, Table 1 · p. 6

4.3 Ablation

• Tool-REX v3 indexing beats name-plus-description alone by 2.2%; appending the raw skill body drops pass rate by a further 3.2%.The authors attribute the drop to implementation detail diluting the discriminative name-and-description signal.Source: §4.3 · p. 6
• With kinit = 20, the cross-encoder performs best; halving candidate depth costs 2.1%, while doubling or quintupling costs 3.5% to 4.5%.The default ablation configuration scored 0.360 here versus 0.409 in the main grid; higher trial parallelism slightly depressed absolute pass rates.Source: §4.3 · pp. 6–7
• Returning k = 1 costs 2.0% relative to k = 5; k = 3, 5, and 10 are tied within 1%, with no added gain beyond k = 3.One or two backup candidates can help when the cross-encoder’s top-ranked choice is wrong.Source: §4.3 · p. 7

4.4 Analysis

• Reranking adds only 3.5% on the 192-skill pool, where stage-1 recall is near ceiling, versus 5.4% on 34K and 9.9% with Tool-REX v3.On 34K, bm25 reaches R@5 = 0.391 and BGE-base reaches 0.379, leaving more room for reranking to improve results.Source: §4.4 · pp. 7–8
• Within the Qwen3-Reranker family, scaling from 0.6B to 8B raises pass rate only from 0.442 to 0.450; 8B beats Voyage rerank-2.5 by 1.5%.Voyage has the highest top-5 recall (72.2%) but a negative helpfulness gap (−0.025), while Qwen3-Reranker-8B’s gap is +0.149; the analysis links this to the agent often committing to rank 1.Source: §4.4 · p. 8

5 Conclusion

• SkillSeek is an open-source two-stage retriever using a BGE-base bi-encoder followed by a small cross-encoder and exposed over MCP.It implements the standard information-retrieval recipe.Source: §5 · p. 9
• Across the 89-task SkillsBench benchmark, SkillSeek reaches observed parity with Liu et al.’s LLM-mediated loop, while per-trial spend drops from $51.30 to $27.54.Plain BM25 suffices on three of four settings, and SkillSeek covers the remaining difference on the fourth.Source: §5 · p. 9
• Under the tested SkillsBench tasks and OpenHands harness, the standard IR recipe is a strong default, with LLM-mediated alternatives fitting cases where it falls short.Source: §5 · p. 9

Limitations

• Each main-grid setting is a single 89-task trial; differences below 2% are treated as within noise, and equivalence requires repeated independent rollouts.The authors report approximately 1% standard deviation across three reruns of one setting.Source: Limitations · p. 10
• Failed trials receive reward 0, lowering every method’s absolute pass rate equally while compressing between-method differences.The authors describe the resulting reported differences as conservative in magnitude, but not in sign.Source: Limitations · p. 10
• MiniMax-M2.7 failures affect roughly half the tasks per condition, so the authors use Qwen3.5-397B-A17B as the headline backbone and MiniMax as supporting evidence.On MiniMax, network or timeout failures lower absolute pass rates by 5% to 10% and narrow method gaps.Source: Limitations · p. 10
• The hand-curated 192-skill pool includes a relevant skill for every task, so its results reflect both retrieval quality and curator selection.The authors make no claim about absolute 192-pool numbers; their finding is the cross-pool pattern.Source: Limitations · p. 10
• First-stage alternatives are compared by R@5 rather than agent pass rate, and higher recall need not yield a more useful top-ranked skill.The authors leave evaluation of the strongest alternatives through the same cross-encoder and agent loop to future work.Source: Limitations · p. 10
• The claim is limited to 89 SkillsBench tasks, one OpenHands harness, two backbones, and two pools; transfer to other agent settings remains open.The authors identify independently built benchmarks such as SkillRet as future transfer tests.Source: Limitations · p. 10

A Model Identifiers

• Model names are shortened in the main text; Table 9 gives canonical identifiers for reproducibility.The harness comparison selects OpenHands SDK because only it combines an in-process Python agent loop with native MCP.Source: A Model Identifiers · p. 11, Table 9 · p. 11, B Choice of Agent Harness · pp. 11–12

C Harness and Implementation Details

• The driver uses a fresh DockerDevWorkspace per trial, caps runs at 30 turns or 600 seconds, and swaps retrievers behind the same skill_lookup tool schema.The MCP server exposes skill_lookup, skill_load, and skill_list; retrieval conditions call lookup on demand.Source: C Harness and Implementation Details · p. 12
• The required first step is a 5-to-15-word skill_lookup query, followed by skill_load before the agent explores the workspace or runs commands.The instruction is included in both the system prompt and first user message.Source: C Harness and Implementation Details · p. 12
• Dual prompting raises skill_lookup invocation to 95.5% on every tested backbone, versus roughly 40% to 60% with the system-only nudge..Source: C Harness and Implementation Details · p. 12

D Baseline Reimplementation

• Every deviation in the reimplemented liu_refined reduces its capabilities relative to the published version, so the accuracy comparison favors SkillSeek while the cost comparison favors the authors’ measurement.The omitted open-ended exploration and unlimited refined-skill output would raise liu_refined’s token spend above the measured $51.30.Source: D Baseline Reimplementation · p. 12
• Exploration is capped at 10 commands and output at 1 to 3 refined skills; on the 34K/Qwen3.5 cell, both Qwen3-Reranker-0.6B and liu_refined score 0.442.The authors say a faithful reimplementation could lift liu_refined above SkillSeek on that cell.Source: D Baseline Reimplementation · p. 13
• On the 192-pool Qwen3.5 cell, Liu et al. report refinement improving pass rate from 0.267 to 0.308 (+4.1%), whereas this reimplementation reports a decline from 0.470 to 0.397 (-7.3%).Liu et al. also report refinement hurting on Kimi K2.5, from 0.335 to 0.267 (-6.8%).Source: D Baseline Reimplementation · p. 13

E Backbone Strength

• MiniMax-M2.7 has roughly 50% to 64% method-independent task failures, limiting method-level interpretation; Qwen3.5 has near-zero such failures.The authors therefore treat Qwen3.5 as the headline backbone and MiniMax as supporting evidence, comparing MiniMax lifts rather than absolute scores.Source: E Backbone Strength · p. 13

F Pilot Experiment on Loading the Entire Pool

• Across 267 trials per condition on the 89 SkillsBench tasks, the pilot compares no skills, all 192 skills, and curated sets of 1 to 3 skills.It uses gpt-oss-120b locally through vLLM and the OpenHands SDK; the authors treat it as supporting evidence, noting that a stronger backbone might absorb more noise from loading all skills.Source: F Pilot Experiment on Loading the… · p. 13

G Tool-REX Profile Iteration

• The authors document the earlier prompt failures as examples of known failure modes in LLM-generated document expansion.All three Tool-REX iterations were generated offline with a locally served gpt-oss-120b; the index is computed once per pool and reused at zero in-loop LLM cost.Source: G Tool-REX Profile Iteration · p. 13
• R@5 on pdf-excel-diff fell from 1.0 to 0.5 when freeform tags omitted the literal surface forms excel and xlsx.The cross-encoder ranked the gold skill at position 2 or 3 instead of 1.Source: G Tool-REX Profile Iteration · p. 13
• At 34K scale, court-form-filling R@5 fell from 1.0 to 0.0 when library-name tags let Python library names dominate the cross-encoder.Those tags also appeared on near-twin skills that were not the gold skill.Source: G Tool-REX Profile Iteration · pp. 13–14
• On the 34K pool with MiniMax-M2.7, Tool-REX v3 lifts agent pass rate by +12.8% over a name+description baseline, with 24 wins, 5 losses, and 41 ties.Its fixed tag order is file-type, primary operation, and two secondaries, with no library-name tags; the profile is generated offline and reused across retrieval trials.Source: G Tool-REX Profile Iteration · pp. 13–14

H Ablation Details

• Replacing Tool-REX v3 with name+description drops pass rate by 2.2%, and appending raw body text drops it by a further percentage point (-3.2%).The body-text loss is consistent with implementation detail diluting the discriminative name+description signal; Tool-REX v3 is the indexing default behind every SkillSeek entry in Table 1.Source: H Ablation Details · p. 14
• At kinit = 20 the reranker performs best; halving depth costs 2.1%, while doubling or quintupling it costs 3.5 to 4.5%.The authors relate this to rank-1 commitment: more cross-encoder input candidates can perturb the top choice, so a clean top ranking matters more than raw recall depth.Source: H Ablation Details · p. 14
• At the agent interface, k = 1 costs 2.0% relative to k = 5, while k = 3, k = 5, and k = 10 are tied within 1%.One or two backups can help when rank 1 is wrong; beyond k = 5, additional candidates buy essentially nothing.Source: H Ablation Details · p. 14

I Helpfulness-Gap Diagnostic: Sign Interpretation

• A positive helpfulness gap means retrieving the gold helps the agent, near-zero means the signal is washed out, and negative means the gold actively confuses it relative to a miss.Voyage rerank-2.5’s ∆ = −0.025 in Figure 2(b) is the clearest negative case in the data.Source: I Helpfulness-Gap Diagnostic: Sign… · p. 14

J Risks

• SkillSeek does not audit, sandbox, or verify surfaced skills, so a malicious or low-quality pool entry can be surfaced and executed in the agent loop.The authors identify a pre-load risk scorer as a natural extension for dropping high-risk candidates before they reach the agent.Source: J Risks · pp. 14–15

K Use of Existing Artifacts

• The evaluation uses the 89-task SkillsBench suite and an approximately 34,000-skill marketplace pool; both pools are English-only and have no train/dev/test split.The SkillsBench pool has 233 SKILL.md files deduplicated to 192 unique names; the marketplace pool had no license file at access time, and neither artifact included an intended-use statement.Source: K Use of Existing Artifacts · p. 15
• A manual audit of 5 SkillsBench tasks and 50 marketplace skills found no significant PII or offensive-content issues, but the sample is too small for population-level claims.The authors describe the audit only as a sanity check on the subset read.Source: K Use of Existing Artifacts · p. 15

L Computational Experiments

• The BGE-base bi-encoder has 110 M parameters and 768-dimensional embeddings, while the default bge-reranker-v2-m3 cross-encoder has 568 M parameters.The reranker sweep also uses Qwen3-Reranker models of 0.6B, 4B, and 8B parameters.Source: L Computational Experiments · p. 15
• Tool-REX v3 index expansion required under 2 GPU-hours on two NVIDIA RTX 6000 GPUs and was computed once per pool for reuse.The same local vLLM stack served gpt-oss-120b for expansion and Qwen3-Embedding-4B for Liu et al. (2026b)’s baseline retriever.Source: L Computational Experiments · p. 15
• SkillSeek’s bi-encoder and cross-encoder run on CPU at approximately 1.1 s median and 1.9 s at p95 per skill_lookup call.The 34K-pool BGE-base embeddings were built once on the same workstation and reused.Source: L Computational Experiments · p. 15
• OpenRouter spend is reported per trial in Table 6 and by backbone in Table 11; the two locally served models are not included in OpenRouter totals.The 89-task commercial-reranker sweep did not exhaust Voyage AI’s 200 million free tokens per account, so its prepaid $50 credit was not drawn down.Source: L Computational Experiments · p. 16, Table 11 · p. 16
• The BM25 baseline uses default rank_bm25 tokenization, and the bi-encoder and cross-encoder use HuggingFace defaults without fine-tuning.Tool-REX v3 uses greedy decoding at temperature 0, with max_tokens capped to 256 per skill.Source: L Computational Experiments · p. 16

M GenAI Disclosure

• The authors report that no AI was used to conceptualize ideas, generate data, or perform evaluations; they used Claude Code for typesetting and LLMs to refine wording.They state that they designed the algorithms, wrote the experimental code, and drafted the manuscript themselves.Source: M GenAI Disclosure · p. 16