PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill2Query: Exploiting Skill Structure to Generate Pseudo-Queries for Agent Skill Retrieval

Lihui Ding, Zihan Guo, Bingwei Lu, Chenyu Zhou, Yuanjian Zhou, Weinan Zhang, Jianghao Lin, Dongdong Ge

arXiv:2608.16071v1cs.CLcs.IR

TL;DR

Existing methods treat skill documents as unstructured text, leaving capability–parameter relations implicit and providing no deterministic guarantee of parameter consistency. Skill2Query parses skill documents into a Skill Knowledge Graph and generates pseudo-queries through style extraction, template generation, and parameter filling. Skill2Query improves retrieval, with R@10 gains of 10.79 pp on LogicBench and 11.88 pp on CHAMP.

  • Problem

    Existing methods treat skill documents as unstructured text, leaving capability–parameter relations implicit and providing no deterministic guarantee of parameter consistency.

  • Method

    Skill2Query parses skill documents into a Skill Knowledge Graph and generates pseudo-queries through style extraction, template generation, and parameter filling.

  • Results

    Skill2Query improves retrieval, with R@10 gains of 10.79 pp on LogicBench and 11.88 pp on CHAMP.

  • Takeaways & Limitations

    Fine-grained capability and parameter correspondence supports more discriminative retrieval matching based on which function a query invokes and which parameters it requires.

  • Takeaways & Limitations

    Performance depends on documentation standardization; missing parameter declarations or usage examples degrade generation quality, and broader applicability remains untested.

Abstract

from arXiv · show

Pseudo-query generation can alleviate the supervision bottleneck for agent skill retrieval, but existing document-level approaches typically leave the rich internal relations among capabilities, parameters, and usage examples implicit. As a result, generated queries may be topically relevant to a skill while lacking capability grounding and parameter consistency, raising the question of whether explicitly exploiting a skill document's internal structure can produce more effective retrieval signals. We therefore propose Skill2Query, a framework that first parses a skill document into a Skill Knowledge Graph and then generates pseudo-queries through a three-stage process including style mimicking, query template generation, and parameter filling. The generated queries can be used for offline index augmentation, online query expansion, and retriever training. Four benchmarks (TheoremQA, LogicBench, ToolQA, and CHAMP) are used to evaluate Skill2Query with large-scale skill candidate pools across multiple downstream applications, including skill retrieval, retriever training, and end-to-end agent execution. Using nearly 30K skills across diverse domains, we generate 700K category-diverse pseudo-queries. Skill2Query consistently improves sparse, dense, and skill-routing retrieval, with an average Recall@1 gain of 6.70 percentage points across retrieval settings. Skill2Query-generated training data also achieves the best Recall@1 and nDCG@1 among the evaluated generation baselines. Further evaluations with multiple LLM backends demonstrate that improved skill retrieval translates into higher agent task success rates. Code and resources are available at https://github.com/MatZaharia/Skill2Query.

1 Introduction

Skill2Query addresses the skill-structure utilization gap in agent skill retrieval by explicitly modeling capabilities, parameters, examples, and their relationships to generate grounded, parameter-consistent pseudo-queries. Across four datasets and three retrieval paradigms, it improves pseudo-query quality and retrieval effectiveness, with an end-to-end case study indicating task-level utility.

  • Problem and Motivation: Skill retrieval is difficult because users issue short, colloquial, and underspecified queries, whereas skill documents use technical descriptions, parameters, constraints, and examples.
  • Problem and Motivation: Existing pseudo-query methods reduce expression mismatch but usually treat skill documents as single text sequences, leaving their internal structure implicit and potentially weakening capability and parameter alignment.
  • Method: The framework parses each skill document into a Skill Knowledge Graph representing capabilities, parameters, examples, and their relationships, then generates queries through style extraction, template generation, and parameter filling.
  • Problem and Motivation: Skill2Query targets the underexplored skill-structure utilization gap by addressing capability grounding and parameter consistency in pseudo-query generation.
  • Contributions: Skill2Query improves pseudo-query quality and retrieval effectiveness across four datasets and three retrieval paradigms, while an end-to-end case study indicates potential task-level utility.

2 Related Work

Prior work shows that agent skill retrieval depends on complete, fine-grained skill information, while pseudo-query generation and structured representations have largely been studied separately. Skill2Query connects these directions by using intra-skill structure to generate parameter-aware pseudo-queries for retrieval augmentation, query expansion, and retriever training.

  • Agent Skill Retrieval and Invocation: Skill retrieval requires complete skill documents containing functional descriptions, execution steps, parameters, constraints, examples, and usage patterns [26].Agent systems must select executable skills from large-scale libraries, making accurate retrieval critical [6] [15].
  • Agent Skill Retrieval and Invocation: Existing skill-retrieval work emphasizes retrieval architectures, while complete skill bodies provide the fine-grained capability, parameter, and constraint information needed for routing.SkillFlow combines dense retrieval, reranking, and LLM selection over approximately 36K SKILL.md files [15], and hiding full skill bodies significantly degrades routing accuracy.
  • Pseudo-Query Generation and Query Expansion: Pseudo-query methods such as doc2query, InPars, HyDE, and Query2doc generate synthetic supervision or expand documents, while tool-learning methods address scarce annotations [4, 23, 1].In agent tool retrieval, related approaches generate from names, descriptions, or examples, or use document expansion and rewriting; Seal-Tools constructs datasets through self-instruct [37] [39].
  • Pseudo-Query Generation and Query Expansion: Flat-text generation loses structural signals from skill parameter schemas, type constraints, capability boundaries, and usage examples, weakening capability grounding and parameter consistency.This limitation is especially consequential for skill retrieval because generators are not explicitly constrained by the document’s internal structure.
  • Structure-Aware Skill Representation: Structured representations improve tool discovery, construction, and invocation reliability, but prior work rarely converts intra-skill structure directly into pseudo-query supervision.Examples include Graph-of-Skills, API knowledge graphs, OpenAPI schemas, and ToolFactory’s structured extraction; Skill2Query parses skill structures via SKG and generates parameter-aware queries for index augmentation, query expansion, and retriever training.

3 Method

Skill2Query represents each skill as a structured Skill Knowledge Graph and uses it to generate capability- and parameter-aware pseudo-queries through style mimicking, template generation, and parameter filling. The resulting pseudo-queries support offline index augmentation, online query expansion, and retriever training.

  • 3.1 Overview: Skill2Query converts each skill document into capability- and parameter-aware pseudo-queries through a three-stage pipeline: style mimicking, query-template generation, and parameter filling.The pipeline uses the Skill Knowledge Graph as a structured intermediate representation.
  • 3.2 Skill Knowledge Graph: The Skill Knowledge Graph explicitly connects skill, capability, parameter, and usage-example nodes, providing a unified representation for subsequent generation stages.Parameter nodes encode names, types, requirement status, defaults, ranges, enumeration constraints, and examples; example nodes retain natural-language queries and corresponding values.
  • 3.3 Pseudo-Query Generation: The generator balances style consistency, capability coverage, and parameter resolvability by extracting example style, producing capability-aware templates, and filling their parameter slots.The style representation includes syntactic patterns, domain vocabulary, parameter expressions, and raw example queries; templates use parameter placeholders and cover varied sentence forms.
  • 3.3 Pseudo-Query Generation: Parameter filling combines candidate values with priority-based sampling and validates parameter names, required coverage, types, ranges, and enumeration constraints.This rule-based validation prevents invalid instantiations and limits combinatorial growth for multi-parameter templates.
  • 3.4 Deployment: Generated pseudo-queries can augment offline indexes, expand online queries with Reciprocal Rank Fusion, or provide synthetic query–skill supervision for retriever training.Each pseudo-query remains associated with its source skill, enabling retrieval of the corresponding skill and construction of training pairs.

4 Experiments and Results

Across four benchmarks and 26,262 candidate skills, Skill2Query improves pseudo-query quality, retrieval, and downstream task success by exploiting structured skill information. Its generated queries provide effective offline and online retrieval augmentation and training data, with gains transferring to end-to-end agent performance.

  • Experimental setup: The evaluation spans TheoremQA, LogicBench, ToolQA, and CHAMP, using 3,160 evaluation instances retrieved from a shared pool of 26,262 skills.Comparisons cover zero-shot, few-shot, SkillFlow-style, and Skill2Query generation, plus BM25, dense retrievers, and SkillRouter under baseline, offline, and online settings.
  • Pseudo-query generation quality: Skill2Query produces stronger parameter validity and functional coverage than baselines, reaching Exec-Pass 42.85% and Func-Coverage 11.32%, while retaining constrained diversity.Exec-Pass improves 16.94 pp over Few-shot, 23.88 pp over Zero-shot, and 20.09 pp over SkillFlow-style; Func-Coverage is 9.11 pp above the best baseline, while Distinct-3 is 85.71%, below Zero-shot and SkillFlow-style.
  • Retrieval performance: Offline augmentation improves most datasets and retrievers, while online query expansion raises BM25 R@1 from 60.37% to 65.19% on TheoremQA and from 13.90% to 18.39% on CHAMP.On CHAMP, online mode also raises BM25 N@10 from 27.91% to 35.60% and SR-Emb-0.6B R@10 from 49.68% to 53.38%; offline and online modes are complementary.
  • Training data quality: Skill2Query training data achieves the best R@1 and N@1 across all four datasets, with larger advantages on LogicBench, ToolQA, and CHAMP.On LogicBench, R@10 reaches 49.34%, improving 10.79 pp over Few-shot and 7.23 pp over SkillFlow-style; on CHAMP, R@10 reaches 62.22% with gains of 5.16 pp and 11.88 pp, while TheoremQA reaches R@1 85.27% and N@10 90.82%.
  • End-to-end task success: Skill2Query-based retrieval improves offline downstream accuracy over no-skill retrieval by 3.48 pp for GPT-5.4, 8.57 pp for DeepSeek-V4-Flash, and 3.34 pp for Qwen3.6-Plus, achieving the strongest non-oracle performance.On GPT-5.4 and Qwen3.6-Plus, Skill2Query-based retrieval reaches 83.53% and 83.13%, within 0.54 and 0.14 pp of Oracle, respectively.

5 Discussion

Skill2Query addresses a perspectival representation gap by separating style, capability structure, and parameter grounding during pseudo-query generation. Its benefits extend to retrieval and agent execution, but depend on documentation quality and require broader validation.

  • Design rationale: Skill2Query bridges developers’ function-complete skill descriptions and users’ task-oriented goals through a shared graph of capabilities, examples, and parameters.The skill knowledge graph provides a common reference frame linking functional atoms, user expression patterns, and executable interfaces.
  • Generation mechanism: Separating information sources improves generation: examples supply style, capability lists provide templates, and schemas constrain parameter values.The reported Exec-Pass gap is 25.91% for Few-shot LLM versus 42.85% for Skill2Query.
  • Retrieval deployment: Offline augmentation and online expansion offer complementary semantic bridging, trading storage and zero query latency against computation latency and contextual adaptability.Offline bridging is precomputed during index construction, whereas online bridging adapts in real time.
  • Training data quality: Skill2Query’s training queries improve relevance granularity by binding each pseudo-query to a concrete capability and a valid parameter space.This fine-grained correspondence distinguishes Skill2Query queries from queries correlated only with overall skill semantics.
  • End-to-end impact: The end-to-end case study preliminarily links retrieval improvements to task-level gains, with stronger retrieval supplementing weaker models and reducing incorrect-skill interference in stronger ones.These observations hold under the evaluated agent configurations.
  • Limitations: Skill2Query’s effectiveness is limited by documentation standardization: missing parameter declarations or usage examples degrade generation toward the Few-shot baseline.Applicability to larger repositories, other agent frameworks, and real user queries remains to be investigated.

6 Conclusion

Skill2Query addresses the scarcity of high-quality query–skill training data by structuring skill information and generating validated pseudo-queries from developer-side definitions. Future work targets personalization, retrieval optimization, transfer, and continual refinement using real interaction data.

  • Conclusion: Skill2Query uses a skill knowledge graph plus template generation, parameter filling, and validity checking to convert skill definitions into pseudo-queries approximating user expressions.The graph organizes skill functions, parameters, and examples.
  • Future Work: Future work will incorporate user history and dialogue context to make pseudo-query generation more personalized and generalizable.
  • Future Work: Future directions include jointly optimizing offline index augmentation with online query expansion, supporting multi-skill retrieval and cross-platform transfer, and refining skill indexes with real interaction data.

A Pseudo-Query Generation Experiment Details

The appendix details the pseudo-query generation pipeline evaluated in RQ1. All methods use the same skill pool and pipeline components, differing only in the information supplied to the query template generator.

  • Experimental Setup: All experiments use the same Skill Pool constructed in this work and share preprocessing, output schema, validation, instantiation, deduplication, and evaluation procedures.The methods are controlled across these components to isolate the effect of the query template generator’s input information.
  • Experimental Setup: The evaluated methods differ only in the input information provided to the query template generator, ranging from plain metadata to full …

A.1 Baseline Methods and Input Construction · A.2 Query Template Generation Prompt · A.3 Parameter Filling and Pseudo-Query Instantiation

The appendix standardizes pseudo-query generation across methods while varying only input information and organization, then uses shared prompting and a three-stage pipeline. Skill2Query uniquely exploits structured capability, parameter, and usage-example information from a parsed SKG throughout generation and instantiation.

  • A.1 Baseline Methods and Input Construction: All methods share the same generation template, parsing, parameter-filling, post-processing, and metric-computation procedures; only input information and organization differ.Table 8 summarizes the compared input configurations.
  • A.1 Baseline Methods and Input Construction: Zero-shot LLM uses skill name and description, Few-shot LLM adds up to three sampled usage examples, SkillFlow-style uses the full unstructured document, and Skill2Query uses a parsed SKG.The SKG organizes capability information, parameter schemas, and usage examples for generation and instantiation.
  • A.2 Query Template Generation Prompt: All approaches reuse the same Query Template Generator, system prompt, output format, and post-processing procedures for fair comparison.The shared query-template system prompt is provided in Table 9.
  • A.2 Query Template Generation Prompt: Under the shared constraints, the generator produces concise, realistic, structurally valid query templates that follow supplied capability and parameter information.Templates contain parameter placeholders for later instantiation.
  • A.3 Parameter Filling and Pseudo-Query Instantiation: The shared QueryGenerationPipeline executes three stages: query-template generation, parameter-slot filling, and pseudo-query expansion.A template may contain slots such as {n} and {k}, which are filled before expansion.
  • A.3 Parameter Filling and Pseudo-Query Instantiation: Zero-shot LLM mainly infers parameter slots from skill names and descriptions, whereas Few-shot LLM additionally uses a small number of example queries.These approaches differ from Skill2Query’s explicit structured parameter guidance.
  • A.3 Parameter Filling and Pseudo-Query Instantiation: Skill2Query guides template generation and filling with SKG parameter nodes and constraints covering types, requiredness, defaults, enumerations, examples, and expression patterns.Concrete slot values are selected according to these parsed constraints.

A.4 Pseudo-Query Generation Quality Metrics … A.4.3 Distinct-3.

Pseudo-query quality is evaluated through parameter validity, functional coverage, and linguistic diversity. The metrics jointly assess constraint satisfaction, capability coverage, and trigram-level variation while distinguishing diversity from correctness.

  • A.4 Pseudo-Query Generation Quality Metrics: Skill2Query evaluates pseudo-query quality using Exec-Pass, Func-Coverage, and Distinct-3 for parameter validity, functional coverage, and linguistic diversity, respectively.
  • A.4.1 Exec-Pass.: Exec-Pass checks whether instantiated query templates satisfy required parameters, parameter types, value ranges, and enumeration constraints.
  • A.4.2 Func-Coverage.: Func-Coverage measures how much of a skill’s capability set is covered by at least one generated query.
  • A.4.2 Func-Coverage.: Overall Func-Coverage averages coverage across skills with non-empty capability sets whose capabilities were successfully parsed.
  • A.4.3 Distinct-3.: Distinct-3 measures linguistic diversity by comparing unique trigrams with the multiset of all trigrams in generated queries.
  • A.4.3 Distinct-3.: Higher Distinct-3 indicates greater diversity in lexical combinations and expression patterns, but not functional correctness or parameter validity.

B Implementation of the online Query Expansion System

The online query expansion system uses GPT-4o-mini to generate two SKG-informed variants per query while preserving the original query, yielding three query views for retrieval. It combines expanded retrieval results through RRF and falls back to baseline retrieval when generation fails.

  • Online query expansion: Two GPT-4o-mini-generated variants are produced per original query from SKG-based seed-skill context for online retrieval.The generator receives the original query and candidate-skill context in a single model call.
  • Online query expansion: The prompt preserves user intent while using candidate-skill capabilities, parameters, examples, templates, and pseudo-queries to normalize concise or incomplete requests.The prompt also prohibits introducing new tasks or tools based on candidate-skill information.
  • Online query expansion: Retaining the original query alongside two generated variants creates three query views for each test instance.The original query is not discarded after expansion, preserving the original semantics.
  • Online retrieval: The expanded retrieval results are reranked by RRF scores, and the fused top-20 skills are returned.This describes the final ranking and selection step of the online retrieval pipeline.
  • Failure handling: After up to two retries, failed generation falls back to original-query retrieval, returning base-retriever results without interrupting or discarding the test instance.The fallback preserves compatibility with the baseline retrieval pipeline and prevents missing evaluation instances.

C Downstream Skill Retrieval Training Setup … C.4 Training Configuration

The downstream study fine-tunes a fixed two-stage SkillRouter pipeline so retrieval differences primarily reflect pseudo-query quality. It defines separate embedding and reranking data, inputs, objectives, and shared training configurations for each generation method.

  • C Downstream Skill Retrieval Training Setup: SkillRouter keeps architectures, input formats, objectives, sampling, and hyperparameters fixed across methods, varying only pseudo-query generation.Each method’s pseudo-queries are paired with corresponding skills and processed through an identical training pipeline.
  • C.1.2 Two-Stage Training Data Construction: Embedding training uses pseudo-query–positive-skill pairs, with skill representations formed from each skill’s name, description, and body.A unique-skill batch sampler prevents instances for the same skill from being treated as in-batch negatives.
  • C.1.2 Two-Stage Training Data Construction: Reranker training uses query-level groups containing one positive and multiple negative skills, derived from top-200 retrieval, deduplication, and reduction to 10 candidates.If necessary, the positive replaces the lowest-ranked candidate, ensuring every group contains exactly one positive skill.
  • C.2.1 Embedding Model Input: The embedding model independently encodes instruction-prefixed queries and skill documents concatenating names, descriptions, and complete bodies.Queries, descriptions, and bodies are truncated to 1,500, 300, and 2,500 characters, respectively, with 2,048-token encoder limits.
  • C.2.2 Reranker Model Input: The reranker jointly encodes each query with a candidate skill document containing its name, description, and body.Reranker inputs allow 1,500-character queries, 500-character descriptions, 2,000-character bodies, and 4,096 tokens.
  • C.3.1 Embedding Objective: Embedding training uses unidirectional query-to-skill in-batch InfoNCE, treating each batch’s matched skill as positive and all other skills as negatives.The temperature parameter is set to 0.05, and similarity uses normalized embeddings with cosine similarity.
  • C.3.2 Reranker Objective: Reranking optimizes listwise cross-entropy over candidate groups using the difference between final-position yes and no logits as relevance score.The temperature is 1.0, directly encouraging the unique positive skill to rank above its group’s candidates.
  • C.4 Training Configuration: Table 11 summarizes the embedding and reranker fine-tuning configurations, which use the same training hyperparameters for every pseudo-query generation method.The configurations provide a shared basis for comparing downstream retrieval performance across generation methods.

D Skill2Query Ablation Study

Ablations show that the Skill Knowledge Graph and parameter-aware generation are complementary: SKG most affects functional coverage, while parameter awareness most directly affects executability. Exec-Verify changes average metrics little but filters invalid outputs before corpus inclusion.

  • w/o SKG: Func-Coverage falls from 11.32% to 2.41% and Exec-Pass from 42.85% to 22.63% without SKG, showing the cost of removing structured skill representations.The SKG organizes Capability, Parameter, and Example information to provide clearer semantic guidance for generation.
  • Overall findings: SKG and parameter-aware generation play complementary roles, with SKG supporting semantic coverage and validity while Param-Aware most directly supports pseudo-query executability.The SKG links capabilities, parameters, and examples, whereas Param-Aware introduces explicit parameter structures into query templates.
  • w/o SKG: Distinct-3 increases from 85.71% to 96.74% without SKG, but the accompanying drops in Exec-Pass and Func-Coverage show that lexical diversity alone does not ensure functional correctness.The increase reflects more dispersed surface forms under weaker structural constraints rather than improved overall generation quality.
  • w/o Param-Aware: Exec-Pass falls from 42.85% to 21.66% without Param-Aware, while Func-Coverage changes from 11.32% to 11.25% and Distinct-3 from 85.71% to 85.54%.Explicit parameter slots and parameter-type information primarily determine whether queries satisfy downstream parsing and execution requirements.
  • w/o Exec-Verify: Exec-Verify changes Exec-Pass from 42.85% to 42.90%, Func-Coverage from 11.32% to 11.27%, and Distinct-3 from 85.71% to 85.62%, with every difference below 0.1 percentage points.Despite limited average-metric gains, verification filters invalid parameter names, incompatible types, out-of-range values, and illegal enumeration values from the final corpus.
Loading 2608.16071v1…