PaperlayerView exact revision on arXiv

Source-linked AI summary

Imagine Before Retrieval: Prospective Skill Retrieval for LLM Agents

Shuo Liu, Yutong Yang, Haohao Xiao, Mouxing Yang, Xi Peng

arXiv:2609.01642v1cs.IR

TL;DR

Agent skill retrieval struggles because objective-oriented task queries do not directly express the procedural capabilities encoded by skills. SkillDreamer addresses this mismatch by inferring capabilities, generating pseudo skills, and incorporating prospective execution requirements into retrieval. Experiments across SkillRet and SkillUsage show gains in retrieval and end-to-end execution, with broader compatibility across retrievers, while generated pseudo skills remain unreliable for direct execution.

  • Problem

    Query–Skill Misalignment separates task objectives from the capabilities and execution guidance encoded by skills, making semantic relevance insufficient for reliable skill identification.

  • Method

    SkillDreamer infers required capabilities, imagines their realization as pseudo skills, and combines these execution requirements with original task semantics during retrieval.

  • Results

    SkillDreamer improves skill retrieval and end-to-end task execution on SkillRet and SkillUsage and generalizes across diverse retrieval models and pipelines.

  • Takeaways & Limitations

    Prospective information can bridge objective-oriented queries and execution-oriented skills, improving alignment between task requirements and executable skills.

  • Takeaways & Limitations

    Generated pseudo skills may contain hallucinated or unverified execution guidelines, so they are unreliable for direct practical use.

Abstract

from arXiv · show

Skill retrieval has recently emerged as a promising paradigm for identifying the desirable execution guidelines from the skill gallery, thus equipping large language model (LLM) agents with the procedural knowledge to accomplish the specified task. To this end, most existing methods customize the retrieval model or reconfigure the retrieval pipeline to prioritize skills that are most semantically relevant to the task query. However, we empirically reveal that task queries and skills are naturally formulated from different perspectives, namely, objective-oriented and procedural-oriented, leading to an under-explored problem termed Query--Skill Misalignment (QSM). Clearly, it is daunting and even impossible to associate the desirable skills in the context of QSM, thus hindering the agent from correctly executing the task. As a remedy, inspired by human prospective cognition, we propose SkillDreamer, a novel framework to alleviate the negative impact of QSM problem. In brief, SkillDreamer first infers the capabilities necessary for task execution, then imagines how to realize these capabilities by generating pseudo skills, and finally leverages such prospective information to bridge the gap between objective-oriented task queries and execution-oriented skills. Extensive experiments on SkillRet and SkillUsage not only verify the effectiveness of SkillDreamer in both skill retrieval and end-to-end task execution, but also demonstrate its generalizability across diverse retrieval models and pipelines. The code will be released upon acceptance.

Introduction

Agent skill retrieval addresses the growing challenge of selecting reusable execution guidance, but Query–Skill Misalignment makes semantic relevance insufficient. SkillDreamer uses prospective reasoning to infer capabilities, imagine realization strategies, and guide retrieval toward executable skills.

  • Motivation: Skill retrieval helps agents identify useful skills from expanding galleries instead of repeatedly solving complex tasks from scratch.The motivation is improved execution efficiency and reliability on complex tasks.
  • Existing Approaches: Existing retrievers mainly prioritize skills semantically relevant to the task query, relying on lexical matching, embeddings, specialized models, reranking, or query decomposition.These approaches refine candidate selection but remain centered on query–skill relevance.
  • Query–Skill Misalignment: Query–Skill Misalignment arises because queries state desired objectives, whereas skills encode capabilities and execution guidance required to achieve them.This mismatch makes semantic relevance insufficient for identifying skills that genuinely support execution.
  • Prospective Retrieval: Prospective cognition motivates inferring indispensable capabilities and imagining how to realize them before retrieving external guidance.The legal-report example illustrates recognizing source-file parsing requirements even when the task does not state them explicitly.
  • SkillDreamer: SkillDreamer combines Capability-aware Inference, Prospective Skill Generation, and Hybrid Skill Retrieval to bridge objective-oriented queries with execution-oriented skills.It preserves semantic anchors, generates pseudo skills describing imagined guidelines, and jointly uses execution requirements with original task semantics.
  • Evaluation: Experiments on SkillRet and SkillUsage verify SkillDreamer for skill retrieval and end-to-end task execution across different retrieval models and pipelines.The framework is presented as plug-and-play and broadly compatible.

Related Work

Prior work adapts information-retrieval methods and retrieval models to identify skills relevant to task queries. This paper argues that agent skills encode executable workflows and resources, creating a Query–Skill Misalignment that standard relevance matching can miss.

  • Information Retrieval: Information retrieval traditionally estimates query–document relevance through lexical matching or learned semantic representations.These methods target relevant information in large-scale corpora.
  • Information Retrieval: Directly transferring information-retrieval paradigms to agent skill retrieval may yield suboptimal performance because skills organize reusable workflows, expertise, and supporting resources.Unlike ordinary documents, skills directly support task execution.
  • Agent Skill Retrieval: Query–Skill Misalignment causes execution-critical skills with weak semantic alignment to be neglected by conventional matching.The mismatch reflects objective-oriented queries versus execution-oriented skills.
  • Agent Skill Retrieval: Agent skill retrieval identifies procedural knowledge from large galleries to facilitate completion of complex tasks.Existing studies are categorized by how they improve matching between task queries and skills.

Method

SkillDreamer addresses Query–Skill Misalignment by inferring task-required capabilities, generating execution-oriented pseudo skills, and integrating them with original-query semantics during retrieval. Its CI, PSG, and HSR components progressively connect objective-oriented queries with procedural skill guidance.

  • Problem Formulation: SkillDreamer formulates QSM as the gap between objective-oriented task queries and execution-oriented skills, which can cause semantically relevant retrievals to miss execution-useful capabilities.Existing retrieval commonly ranks skills by semantic relevance to the query, but QSM separates query objectives from the capabilities and guidance encoded in skills.
  • Capability-aware Inference: The CI module infers capability-level subqueries from the task while preserving task-specific semantic anchors from the original query.Capability inference excludes trivial operations and retains cues such as domain terminology and key task targets.
  • Prospective Skill Generation: PSG converts each inferred capability into a pseudo skill that describes how the capability could be realized through concrete procedural guidance.These pseudo skills act as retrieval proxies rather than executable skills because generated guidelines may be hallucinated or unverified.
  • Hybrid Skill Retrieval: HSR jointly considers original-query relevance and capability requirements when constructing candidate skill sets.The method embeds the query, pseudo skills, and gallery skills, then fuses query relevance with pseudo-skill relevance using a weighted score.
  • Hybrid Skill Retrieval: Semantic anchors strengthen candidate selection by providing BM25-based reranking cues tied to the original task semantics.The anchors are concatenated into a lexical query and used to rerank candidates within each capability-specific set.
  • Hybrid Skill Retrieval: HSR combines the capability-specific ranked lists in round-robin order, removes duplicates, and stops after obtaining K unique skills.The ranking matrix uses capabilities as columns and ranking positions as rows before scanning the lists to produce the final ordered retrieval list.

Experiments

Experiments evaluate SkillDreamer across retrieval benchmarks, retrievers, and end-to-end agents, showing consistent retrieval and execution gains while testing component contributions and robustness.

  • Experimental setup: SkillDreamer is evaluated on SkillRet and SkillUsage with multiple retriever paradigms and end-to-end task execution settings.SkillUsage supports retrieval and execution evaluation, while SkillRet evaluates large-scale skill retrieval.
  • Skill retrieval: 7.74% and 7.79% R@5 gains result from applying SkillDreamer to Qwen3-Embedding-0.6B on SkillUsage and SkillRet, respectively.The gains are reported across the two benchmarks and support generalization across retrieval backbones.
  • Skill retrieval: 56.15% R@5 is achieved after SkillDreamer improves SkillRouter from 46.55% on SkillUsage.The result shows that stronger semantic matching alone does not necessarily produce better skill retrieval.
  • End-to-end execution: 16.1% and 11.0% pass-rate improvements are obtained for MiniMax-M3 and DeepSeek-V4-Flash versus no-skill settings.SkillDreamer consistently improves task execution across the evaluated agents.
  • End-to-end execution: Pseudo skills have higher load rates but lower pass rates than retrieved skills, so SkillDreamer uses them as a retrieval bridge rather than directly for execution.The paper attributes the lower pass rates to unverified and potentially unreliable self-generated procedures.
  • Ablation and analysis: Ablation studies compare raw queries, semantic anchors, inferred capabilities, and other components to assess their contributions to SkillDreamer.The study is conducted on SkillUsage with Qwen3-Embedding-0.6B unless otherwise specified.

Conclusion

The paper identifies Query–Skill Misalignment between objective-oriented queries and execution-oriented skills, then proposes SkillDreamer to address it through prospective retrieval. Experiments support its effectiveness and generalizability for retrieval and end-to-end execution.

  • Conclusion: Query–Skill Misalignment arises because task queries describe user objectives while skills provide capabilities and execution guidance.The paper argues that semantic relevance alone is insufficient for identifying skills that genuinely support task completion.
  • Conclusion: SkillDreamer uses human prospective cognition to anticipate required capabilities and imagine realization strategies before retrieval.The framework is designed to integrate with existing retrieval approaches.
  • Conclusion: Experiments demonstrate SkillDreamer’s effectiveness and generalizability in skill retrieval and end-to-end task execution.The conclusion reports these findings on representative benchmarks.
  • Conclusion: Future work will extend SkillDreamer to scenarios where agents dynamically adjust skill requirements using environmental feedback during execution.This defines the paper’s stated future scope.

Supplementary Material for Imagine Before Retrieval: Prospective Skill Retrieval for LLM Agents

The listed authors are Shuo Liu, Yutong Yang, Haohao Xiao, Mouxing Yang, Xi Peng.

  • The author list names Shuo Liu and Yutong Yang as co-first authors.
  • The author list also includes Haohao Xiao, Mouxing Yang, and Xi Peng.
  • Xi Peng is affiliated with superscripted affiliations 2 and 3, while the other listed authors carry affiliation 1.

Introduction

The supplied passage identifies the supplementary material as a source of further implementation details and experimental results for SkillDreamer.

  • Supplementary material: The supplementary material provides further implementation details and additional experimental results for SkillDreamer.The passage describes these materials as supporting evidence for the framework’s effectiveness.

Additional Details of Implementation

SkillDreamer combines capability inference, pseudo-skill generation, and hybrid retrieval, and evaluates the framework on two datasets using retrieval and execution metrics.

  • Retrieval procedure: Hybrid retrieval combines original query similarity with pseudo-skill similarity, then reranks candidates using anchor relevance before merging capability-specific lists.The procedure retrieves top-Kc candidates, reranks top-Kr candidates, and selects the final top-K skills through round-robin merging.
  • Datasets: SkillDreamer is evaluated on SkillRet, built from public open-source skills, and SkillUsage, which pairs executable tasks with corresponding skills.Both benchmarks support skill-retrieval evaluation, while SkillUsage additionally represents executable real-world tasks.
  • Framework prompts: The CI module infers retrieval-oriented capability requirements while preserving evidence from the original query.Its prompt is designed to produce self-contained capability requirements anchored in the task semantics.
  • Framework prompts: The PSG module converts inferred capabilities and preserved evidence into concise hypothetical SKILL.md documents containing solution-oriented knowledge.These pseudo skills provide prospective execution guidance for subsequent retrieval.
  • Experimental settings: The default settings use Kc = 20, Kr = 20, and K = 15, with α = 2, β = 1, and γ = 1 for most embedding backbones.SkillRet-Embedding-0.6B and R3-Embedding on SkillRet instead use α = 4, β = 1, and γ = 0.25.
  • Evaluation metrics: Evaluation covers Recall@K, NDCG@K, Completeness@K, Pass Rate, and Load Rate across retrieval and end-to-end task execution.Recall measures retrieved ground-truth skills, NDCG incorporates rank, Completeness requires all ground-truth skills, Pass Rate averages verifier rewards, and Load Rate tracks whether skills are loaded.

Additional Experiments

Additional experiments show that SkillDreamer improves end-to-end execution and retrieval quality across datasets, benchmarks, retriever backbones, and task complexities, while remaining stable across α values.

  • Unified prospective retrieval and execution: SkillDreamer raises Pass Rate from 31.3% to 37.6% over Raw Query Retrieval in unified Qwen3.6-27B execution.The comparison uses the same QCoder harness for prospective reasoning, retrieval, and task execution.
  • Additional retrieval results: On SkillUsage and SkillRet with Qwen3-Embedding-0.6B, NDCG@5 improves by 4.79 and 4.68 points, while Completeness@5 improves by 6.90 and 9.76 points, respectively.The gains extend beyond ranking quality to recovery of complete required skill sets.
  • Generalization to SRA-Bench: On SRA-Bench, Recall@5 increases from 57.90 to 66.08, NDCG@5 from 52.93 to 61.58, and Completeness@5 from 51.43 to 59.19.The reported gains are 8.18, 8.65, and 7.76 points, respectively, with similar improvements at K = 10 and K = 15.
  • Retriever generalization: SkillDreamer consistently improves Recall, NDCG, and Completeness across BGE-large-en-v1.5, E5-large-v2, and Qwen3-Embedding-8B.These experiments test plug-and-play use across additional embedding backbones on SkillUsage and SkillRet.
  • Sensitivity analysis: SkillDreamer performs stably under varying α values on SkillUsage, indicating robustness to the original-query weight.In the hybrid retrieval stage, α controls the contribution of the original query.
  • Skill-count analysis: As the number of ground-truth skills increases, SkillDreamer increasingly outperforms raw query retrieval across R@5, R@10, and R@15.The larger gains on multi-skill queries indicate stronger performance for tasks requiring diverse skills.
Loading 2609.01642v1…