Imagine Before Retrieval: Prospective Skill Retrieval for LLM Agents

Introduction

• As skill galleries expand, retrieval aims to find useful skills for each task; existing approaches generally rank skills by semantic relevance to the query.Prior approaches include lexical or embedding retrieval, specialized retrievers, retrieve-and-rerank pipelines, and query decomposition.Source: Introduction · pp. 1–2
• QSM causes semantically relevant skills to outrank execution-critical skills when objective-oriented queries emphasize task background, degrading downstream task execution.For a legal-report task, legal-oriented skills may be prioritized while a needed PDF-processing skill is neglected.Source: Introduction · p. 2
• Inspired by prospective cognition, SkillDreamer infers indispensable capabilities and imagines how to realize them before searching the skill gallery.This is intended to expose requirements that the task does not explicitly state and bridge objective-oriented queries with execution-oriented skills.Source: Introduction · p. 2
• SkillDreamer combines capability inference, prospective pseudo-skill generation, and hybrid retrieval to jointly account for task relevance and execution requirements.CI uncovers expected capabilities while preserving semantic anchors; PSG imagines guidelines for realizing those capabilities; HSR integrates them with the original task semantics.Source: Introduction · p. 2
• Experiments on SkillRet and SkillUsage verify effectiveness in skill retrieval and end-to-end execution, while plug-and-play gains across retrieval models demonstrate broad compatibility.The authors frame QSM as a shift from semantic matching alone toward jointly capturing task relevance and capability requirements.Source: Introduction · p. 2

Information Retrieval

• Information retrieval methods use lexical or learned semantic matching, while LLM-assisted approaches enhance queries before retrieval or refine candidate relevance afterward (Robertson and Zaragoza 2009; Karpukhin et al. 2020; Ma et al. 2023; Wang, Yang, and Wei 2023; Gao et al. 2023; Sun et al. 2023; Qin et al. 2024; Reddy et al. 2024).Source: Information Retrieval · p. 2
• Skills package reusable workflows and expertise, so semantically weakly aligned but execution-critical skills can be neglected under conventional document-relevance retrieval, motivating SkillDreamer for QSM.The paper frames QSM as task queries specifying objectives while skills express capabilities and execution guidance.Source: Information Retrieval · pp. 2–3

Agent Skill Retrieval

• Agent skill retrieval adapts models or redesigns pipelines, whereas SkillDreamer uses prospective reasoning before retrieval to infer capability requirements and bridge objective-oriented queries with execution-oriented skills (Cho, Kang, and Kim 2026; Wang et al. 2026; Zheng et al. 2026; Li, Tagkopoulos, and Tagkopoulos 2026; Chen et al. 2026; Zeng et al. 2026; Ding 2026).Source: Agent Skill Retrieval · p. 3

Problem Formulation

• Semantic similarity can prioritize skills that are not useful for task execution, creating Query–Skill Misalignment between task queries and execution-useful skills.The method frames QSM as a failure of semantic relevance alone to prioritize skills that support task completion.Source: Problem Formulation · p. 3

Capability-aware Inference

• Capability-aware Inference uses an LLM to infer implicit execution capabilities, excludes trivial operations, and extracts semantic anchors to retain task-specific context.The resulting capabilities add execution-relevant information, while anchors preserve cues such as domain terminology and task targets.Source: Capability-aware Inference · pp. 3–4

Prospective Skill Generation

• Prospective Skill Generation converts each inferred capability into a skill-like pseudo skill describing concrete procedural guidance, which serves as a retrieval proxy rather than verified guidance.The passage cautions that generated pseudo skills may contain hallucinated or unverified instructions; their purpose is to identify similar gallery skills.Source: Prospective Skill Generation · p. 4, Hybrid Skill Retrieval · pp. 4–5

Hybrid Skill Retrieval

• Hybrid Skill Retrieval combines original-query relevance with pseudo-skill capability relevance, reranks candidates using preserved semantic anchors, then round-robins capability-specific lists while removing duplicates.This lets each inferred capability contribute highly ranked skills to the final list.Source: Hybrid Skill Retrieval · pp. 4–5

Experimental Setup

• Evaluation uses SkillRet’s 4,997 queries and 6,660 skills, and 84 valid SkillUsage tasks with 34,396 candidate skills, reporting Recall@K and execution Pass and Load.The default retriever is Qwen3-Embedding-0.6B; GPT-5.5, MiniMax-M3, and DeepSeek-V4-Flash receive the top five retrieved skills for end-to-end evaluation.Source: Experimental Setup · p. 5

Experiments on Skill Retrieval

• On SkillUsage and SkillRet, SkillDreamer consistently improves retrieval across benchmarks and backbones; on Qwen3-Embedding-0.6B, it raises R@5 by 7.74% and 7.79%, respectively.Source: Experiments on Skill Retrieval · p. 5, Table 1 · p. 6

Experiments on End-to-End Task Execution

• On SkillUsage, SkillDreamer raises MiniMax-M3 and DeepSeek-V4-Flash pass rates by 16.1% and 11.0% over no-skill settings, respectively.Pseudo skills achieve higher load rates than retrieved skills for these two agents but lower pass rates, supporting their use as a bridge rather than direct execution procedures.Source: Table 2 · p. 6, Experiments on End-to-End Task Execution · pp. 6–7

Experiments on Skill Retrieval

• SkillRouter’s SkillUsage R@5 is below the base model’s, while SkillDreamer raises it from 46.55% to 56.15%, showing stronger semantic matching alone is insufficient.The authors attribute the limitation to the mismatch between objective-oriented queries and execution-oriented skills.Source: Experiments on Skill Retrieval · p. 6

Ablation and Analytic Studies

• Ablations show capability decomposition, capability inference before pseudo-skill generation, combining task semantics with requirements, and anchor reranking each contribute to retrieval performance.CI-C + PSG consistently outperforms PSG, and the full method’s anchor reranking complements the combined original-query and pseudo-skill retrieval results.Source: Table 3 · p. 7, Ablation and Analytic Studies · p. 7
• SkillDreamer’s retrieval performance remains stable under substantial random displacement of pseudo-skill sentences, indicating an exact execution pathway is unnecessary.The study measures performance across sentence displacement rates using three random permutations.Source: Figure 3 · p. 7, Ablation and Analytic Studies · p. 7
• SkillDreamer outperforms raw-query retrieval across SkillUsage task groups, retaining gains as the number of ground-truth skills increases.The analysis groups tasks by their number of ground-truth skills to assess retrieval under varying capability requirements.Source: Figure 4 · p. 7, Ablation and Analytic Studies · p. 7

• Query–Skill Misalignment arises because task queries describe user objectives while skills provide capabilities and execution guidance, making semantic relevance insufficient to identify useful skills.Source: Conclusion · p. 7
• SkillDreamer uses prospective cognition to address this mismatch and can be integrated into existing approaches to improve skill retrieval performance.Source: Conclusion · p. 7
• Experiments on representative benchmarks demonstrate the necessity of addressing QSM and verify SkillDreamer’s effectiveness and generalizability in skill retrieval and end-to-end task execution.Source: Conclusion · p. 7
• Future work will extend SkillDreamer to complex scenarios where agents dynamically adjust skill requirements based on environmental feedback during execution.Source: Conclusion · p. 7

• The supplementary material adds implementation details and experimental results intended to demonstrate SkillDreamer’s effectiveness.Source: Introduction · p. 9

Details of the Algorithm

• SkillDreamer infers capabilities and anchors, generates pseudo-skills, combines query–skill and pseudo-skill–skill similarities, reranks with BM25 anchor relevance, then round-robin merges capability-specific lists.It retrieves Kc candidates per capability, reranks Kr candidates, and returns the top K skills.Source: Details of the Algorithm · p. 9

Details of the Datasets

• SkillRet uses LLM-generated queries from sampled public skills as retrieval targets, while SkillUsage pairs executable tasks with corresponding skills in a large real-world skill library.SkillUsage is built upon SkillsBench.Source: Details of the Datasets · p. 9

Details of the Prompts

• The CI prompt elicits self-contained, retrieval-oriented capabilities while preserving query evidence; the PSG prompt turns these into concise hypothetical SKILL.md documents with solution-oriented knowledge.The prompts are provided in Tables 5 and 6, respectively.Source: Details of the Prompts · p. 9

Details of the Baselines

• Baselines include BM25 sparse retrieval, general embedding retrievers Qwen3-Embedding, BGE-large-en-v1.5, and E5-large-v2, and skill-oriented SkillRouter-Embedding-0.6B, SkillRet-Embedding-0.6B, and R3-Embedding.Their training strategies include in-batch or multi-positive InfoNCE, with R3-Embedding also using a sibling-reward term to preserve multi-skill relevance.Source: Details of the Baselines · pp. 9, 11

Details of the Experimental Settings

• Default retrieval scales are Kc = 20, Kr = 20, and K = 15, with α = 2, β = 1, and γ = 1; SkillRet-Embedding-0.6B and R3-Embedding on SkillRet instead use α = 4, β = 1, and γ = 0.25.Kc is the initial hybrid-score candidate count, Kr the reranking candidate count, and K the final returned list size.Source: Details of the Experimental Settings · p. 11

Details of the Evaluation Metrics

• Retrieval evaluation uses Recall@K, NDCG@K for ranking quality, and Completeness@K for retrieving all ground-truth skills within the top K.NDCG@K gives greater gains to relevant skills at higher ranks.Source: Details of the Evaluation Metrics · p. 11
• End-to-end execution is evaluated by Pass Rate, the average verifier reward across tasks, and Load Rate, the proportion of tasks where the agent loads at least one provided skill.Missing or invalid verifier rewards count as zero.Source: Details of the Evaluation Metrics · p. 11

Unified Prospective Retrieval and Execution

• 37.6% versus 31.3% Pass rate: SkillDreamer outperforms Raw Query Retrieval in unified Qwen3.6-27B execution.Prospective reasoning infers capabilities and procedural guidance; externalized workflows and pseudo skills can help the agent address overlooked operational details and constraints.Source: Table 7 · p. 11, Unified Prospective Retrieval and… · pp. 11–12

Additional Retrieval Results

• On Qwen3-Embedding-0.6B, SkillDreamer improves NDCG@5 by 4.79 and 4.68 points and Completeness@5 by 6.90 and 9.76 points on SkillUsage and SkillRet, respectively.Consistent gains across backbones and improved Completeness@K indicate recovery of missing required skills, beyond merely reordering candidates.Source: Additional Retrieval Results · p. 12, Table 8 · p. 12, Table 9 · p. 13

Generalization across Diverse Backbones

• SkillDreamer consistently improves Recall, NDCG, and Completeness across BGE-large-en-v1.5, E5-large-v2, and Qwen3-Embedding-8B.The results support plug-and-play integration across embedding backbones without dependence on a specific model.Source: Generalization across Diverse Backbones · p. 12, Table 11 · p. 14, Table 12 · p. 14

Generalization to SRA-Bench

• On SRA-Bench with Qwen3-Embedding-0.6B, SkillDreamer raises Recall@5 from 57.90 to 66.08, NDCG@5 from 52.93 to 61.58, and Completeness@5 from 51.43 to 59.19.Gains were also observed at K = 10 and K = 15.Source: Generalization to SRA-Bench · p. 12, Table 10 · p. 13

Parameter Analysis

• SkillDreamer performs stably across varying original-query weights α on SkillUsage, demonstrating robustness to this parameter choice.In Hybrid Skill Retrieval, α controls the original query’s contribution.Source: Parameter Analysis · p. 13, Figure 5 · p. 13

Additional Analysis on the Number of Skills

• As ground-truth skill counts increase on SkillRet, SkillDreamer increasingly outperforms raw query retrieval across R@5, R@10, and R@15.Performance is comparable for single-skill queries, while gains grow for more complex multi-skill queries.Source: Additional Analysis on the Number of… · p. 13