PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill2Query: Exploiting Skill Structure to Generate Pseudo-Queries for Agent Skill Retrieval

Lihui Ding, Zihan Guo, Bingwei Lu, Chenyu Zhou, Yuanjian Zhou, Weinan Zhang, Jianghao Lin, Dongdong Ge

arXiv:2608.16071v1cs.CLcs.IR

TL;DR

Existing pseudo-query methods often treat skill descriptions as unstructured text, leaving capability–parameter relations implicit. Skill2Query structures skill documents in a Skill Knowledge Graph before generating capability- and parameter-aware pseudo-queries; experiments report gains in pseudo-query quality and retrieval effectiveness across four datasets and three retrieval paradigms.

  • Problem

    Existing pseudo-query methods treat skill descriptions as flat text, leaving capability–parameter relations implicit despite their importance to skill retrieval.

    Source: §2.2 · p. 3

  • Method

    Skill2Query parses skill documents into a Skill Knowledge Graph and uses a three-stage pipeline to generate capability- and parameter-aware pseudo-queries.

    Source: §3 · p. 3

  • Results

    Experiments across four datasets and three retrieval paradigms report improvements in pseudo-query quality and retrieval effectiveness, while an end-to-end case study indicates potential task-level utility.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The conclusion reports that Skill2Query also improves agents’ end-to-end success rates on real tasks.

    Source: §6 · p. 9

    Generation quality depends on documentation standardization and degrades toward the Few-shot baseline when parameter declarations or usage examples are absent.

    Source: §5 · p. 8

Abstract

from arXiv · show

Pseudo-query generation can alleviate the supervision bottleneck for agent skill retrieval, but existing document-level approaches typically leave the rich internal relations among capabilities, parameters, and usage examples implicit. As a result, generated queries may be topically relevant to a skill while lacking capability grounding and parameter consistency, raising the question of whether explicitly exploiting a skill document's internal structure can produce more effective retrieval signals. We therefore propose Skill2Query, a framework that first parses a skill document into a Skill Knowledge Graph and then generates pseudo-queries through a three-stage process including style mimicking, query template generation, and parameter filling. The generated queries can be used for offline index augmentation, online query expansion, and retriever training. Four benchmarks (TheoremQA, LogicBench, ToolQA, and CHAMP) are used to evaluate Skill2Query with large-scale skill candidate pools across multiple downstream applications, including skill retrieval, retriever training, and end-to-end agent execution. Using nearly 30K skills across diverse domains, we generate 700K category-diverse pseudo-queries. Skill2Query consistently improves sparse, dense, and skill-routing retrieval, with an average Recall@1 gain of 6.70 percentage points across retrieval settings. Skill2Query-generated training data also achieves the best Recall@1 and nDCG@1 among the evaluated generation baselines. Further evaluations with multiple LLM backends demonstrate that improved skill retrieval translates into higher agent task success rates. Code and resources are available at https://github.com/MatZaharia/Skill2Query.

1 Introduction

Skill2Query addresses the skill-structure utilization gap by generating pseudo-queries grounded in capabilities and consistent with their parameters. Evaluations report improved pseudo-query quality and retrieval effectiveness, with an end-to-end case study indicating potential task-level utility.

1 Introduction

  • Skill retrieval matches short, colloquial, underspecified user queries [18] against developer-written skill documents, and obtaining high-quality query–skill supervision through manual annotation is expensive [1] [8] [35].Retrieval errors can expose agents to irrelevant capabilities or exclude skills needed to complete tasks [32].Source: §1 · p. 1
  • Document-level generation can produce queries related to a skill without grounding them in a specific capability or keeping parameter values consistent with its interface and constraints [1] [8] [23] [35].The paper frames these limitations as capability-grounding and parameter-consistency gaps, despite generators having access to the full skill document.Source: §1 · p. 2
  • Skill2Query builds a Skill Knowledge Graph and uses style mimicking, capability-linked query templates, and validated parameter filling to generate queries for index augmentation, query expansion, and retriever training.The graph represents capabilities, parameters, examples, and their relationships; style patterns come from usage examples, and parameter slots are instantiated according to parameter definitions.Source: §1 · p. 2
  • Experiments across four datasets and three retrieval paradigms show improved pseudo-query quality and retrieval effectiveness, while an end-to-end case study indicates potential task-level utility.The introduction does not specify quantitative results for these improvements.Source: §1 · p. 2

2 Related Work

Related work positions Skill2Query at the intersection of agent skill retrieval, pseudo-query supervision, and structure-aware representation, addressing the gap between these previously separate research directions.

2.1 Agent Skill Retrieval and Invocation

  • Skill retrieval must match user intent to capability boundaries and executable conditions; hiding full skill bodies degrades routing accuracy [15] [43].Prior work has focused on retrieval architectures rather than systematically addressing the source of high-quality query–skill training data.Source: §2.1 · p. 2

2.2 Pseudo-Query Generation and Synthetic Supervision

  • Synthetic queries can support retrieval indices and retriever training, but flat-text generation loses skill structure and cannot guarantee parameter consistency.Prior methods also use synthetic supervision for tool retrieval and learning, often generating queries from tool names, descriptions, or examples.Source: §2.2 · p. 3

2.3 Structure-Aware Skill Representation and Retrieval

  • Prior structure-aware methods model inter-skill relationships, while Skill2Query parses intra-skill metadata, capabilities, parameters, and examples into parameter-aware pseudo-queries for indexing, expansion, and training.Graph-of-Skills models inter-skill dependencies [19], while structured API representations support tool discovery, construction, and invocation reliability [11] [22].Source: §2.3 · p. 3

3 Method

Skill2Query represents skills with a Skill Knowledge Graph and uses a three-stage pipeline to generate pseudo-queries grounded in capabilities and parameters. Its design separates style consistency, capability coverage, and valid parameter instantiation.

3.1 Problem Formalization

  • Retrieval seeks skills that match both a query’s task intent and the consistency of its implied arguments with each skill’s interface.Source: §3.1 · p. 3

3.2.1 Node Types.

  • The graph defines Skill, Capability, Parameter, and Example nodes; parameter attributes encode constraints and values, while examples pair natural-language queries with parameters.Parameter fields include name, type, required status, default, range, enumeration constraints, and examples.Source: §3.2.1 · p. 4

3.2 Skill Knowledge Graph

  • The Skill Knowledge Graph explicitly connects capabilities, parameters, and usage examples into a unified representation for subsequent query generation.Source: §3.2 · p. 4, §3.2.1 · p. 5

3.3 Three-stage Progressive Generation Architecture

  • The three-stage pipeline decouples style consistency, capability coverage, and parameter resolvability into progressive transformations: Style Mimicker, Query Template Generator, and Param Filler.The method uses separate stages because a single LLM generation process is difficult to optimize all three objectives simultaneously.Source: §3.3 · p. 5

3.3.1 Style Mimicker.

  • The Style Mimicker extracts syntactic patterns, domain vocabulary, parameter expressions, and raw example queries as style references for template generation.Source: §3.3.1 · p. 5, §3.3.2 · p. 5

3.3.2 Query Template Generation.

  • Templates associate natural-language text with a capability and parameter slots, while covering capabilities and varying sentence forms.Forms include imperatives, yes/no questions, wh-questions, and indirect requests; the number of templates per skill is bounded by B.Source: §3.3.2 · p. 5

3.3.3 Parameter Filling.

  • The Param Filler instantiates slots from parameter definitions and example expressions, then statically validates names, required parameters, types, ranges, and enumeration constraints.For multi-parameter templates, it combines candidate values by Cartesian product and caps combinations using priority-based sampling.Source: §3.3.3 · p. 5

3.4 Deployment of Generated Pseudo-Queries

  • Generated pseudo-queries support offline index augmentation, online query expansion with Reciprocal Rank Fusion, and synthetic query–skill supervision for retriever training.Source: §3.4 · p. 6

4 Experiments and Results

Skill2Query improves retrieval through offline index augmentation and query-conditioned online expansion, while its pseudo-queries also provide strong training data and downstream gains in end-to-end answering. Across four benchmarks, it is evaluated against several generation and retrieval baselines using a shared pool of 26,262 skills.

4.1 Experimental Setup

  • Evaluation covers TheoremQA [3], LogicBench [27], ToolQA [45], and CHAMP [20], with 3,160 instances retrieved from a shared pool of 26,262 skills.Comparisons assess generation quality, sparse, dense, and two-stage retrieval in baseline, offline, and online modes, and retrieval training data quality.Source: §4.1 · p. 6

4.2 RQ1: Pseudo-Query Generation Quality

  • Skill2Query reaches 42.85% Exec-Pass and 11.32% Func-Coverage, while its Distinct-3 of 85.71% exceeds Few-shot but trails Zero-shot and SkillFlow-style.Exec-Pass improves by 16.94 pp over Few-shot, 23.88 pp over Zero-shot, and 20.09 pp over SkillFlow-style; Func-Coverage is 9.11 pp above the best baseline.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3 RQ2: Retrieval Performance Gains

  • On ToolQA, offline augmentation raises BM25 R@1 from 5.52% to 32.03% and improves R@1 by 15.11 pp for BGE and 21.75 pp for SR-Emb-0.6B.SkillRouter R@1 rises from 35.80% to 47.34%; R@10 gains often exceed R@1 gains, and augmentation is especially effective with skill names or brief descriptions.Source: §4.3 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • Online expansion raises BM25 R@1 on TheoremQA from 60.37% to 65.19% and on CHAMP from 13.90% to 18.39%, complementing offline index supplementation with query-specific variants.On CHAMP, BM25 N@10 improves from 27.91% to 35.60%, while SR-Emb-0.6B R@10 rises from 49.68% to 53.38%.Source: §4.3 · p. 6, Table 3 · p. 7, Table 4 · p. 7

4.4 RQ3: Quality of Pseudo-Query Training Data

  • Skill2Query training data achieves the best R@1 and N@1 across all four benchmarks, with larger advantages on LogicBench, ToolQA, and CHAMP.On LogicBench, R@10 is 49.34% and N@10 is 34.84%; on CHAMP, R@10 is 62.22% and N@10 is 52.22%.Source: §4.4 · pp. 7–8, Table 5 · p. 7, Table 6 · p. 7

4.5 Case Study: End-to-End Task Performance Validation

  • In end-to-end TheoremQA, fine-tuned SkillRouter improves accuracy over no-skill by 3.48 pp for GPT-5.4, 8.57 pp for DeepSeek-V4-Flash, and 3.34 pp for Qwen3.6-Plus.Skill2Query retrieval reaches 83.53% for GPT-5.4 and 83.13% for Qwen3.6-Plus, respectively 0.54 and 0.14 pp below Oracle; DeepSeek-V4-Flash rises from 73.76% to 82.33%.Source: §4.5 · p. 8, Table 7 · p. 8

5 Discussion

The discussion attributes Skill2Query’s retrieval and task-level benefits to structurally grounded query generation, while identifying documentation quality and broader applicability as important limitations.

Discussion

  • Skill documents describe what developers can provide, while users express what they want to accomplish; the SKG links these perspectives through Capability, Example, and Parameter nodes.Capabilities capture functional atoms, Examples capture user-expression patterns, and Parameters define the executable interface between them.Source: §5 · p. 8
  • Skill2Query achieved 42.85% Exec-Pass versus 25.91% for Few-shot LLM, which mixes information sources rather than assigning each generation stage its relevant source.Style comes from examples, template structure from capability lists, and parameter values from schema definitions.Source: §5 · p. 8
  • On TheoremQA, BM25 full offline augmentation yielded no gain, while online expansion raised R@1 from 60.37% to 65.19%.Offline augmentation trades storage for zero query latency; online expansion trades computation latency for contextual adaptability, and static supplementation may add less when skill documents already contain rich semantic signals. The optimal choice may depend on document semantic richness.Source: §5 · p. 8
  • Skill2Query training queries produced R@10 gains of 10.79 pp on LogicBench and 11.88 pp on CHAMP, consistent with their higher skill complexity.Unlike queries tied only to overall skill semantics, each pseudo-query maps to a concrete Capability and valid parameter space, encouraging more discriminative retrieval patterns.Source: §5 · p. 8
  • The end-to-end case study offers preliminary evidence that retrieval improvements can raise task success, with greater supplementation for weaker models and less interference from incorrect skills on stronger ones [21].When stronger models’ retrieval approaches Oracle, it reduces interference from incorrect retrieved skills.Source: §5 · p. 8
  • Skill2Query’s generation quality degrades toward the Few-shot baseline when documents lack parameter declarations or usage examples, and its efficacy ceiling depends on documentation standardization.Applicability to larger repositories, other agent frameworks, and real user queries remains to be investigated.Source: §5 · pp. 8–9

6 Conclusion

Skill2Query structures skill functions, parameters, and examples in a knowledge graph to generate pseudo-queries, improving retrieval across multiple settings and agent task success. Future work targets personalization, joint retrieval and expansion, cross-platform transfer, and continual refinement from real interactions.

6 Conclusion

  • Skill2Query uses a skill knowledge graph, template generation, parameter filling, and validity checking to turn developer-side definitions into user-like pseudo-queries that improve retrieval and end-to-end agent success.The conclusion reports improvements in pseudo-query quality, offline index augmentation, online query expansion, and retrieval model training.Source: §6 · p. 9
  • Future work will incorporate user history and dialogue context, co-optimize offline and online retrieval, investigate multi-skill retrieval and cross-platform transfer, and refine indexing and training data with real interactions.These directions aim to improve personalization and generalization and support more efficient, scalable large-scale agent skill retrieval.Source: §6 · p. 9

A Pseudo-Query Generation Experiment Details

The appendix describes the controlled setup for the RQ1 pseudo-query-generation experiment, in which methods differ only in the information given to the template generator.

A Pseudo-Query Generation Experiment Details

  • Methods share the Skill Pool and pipeline controls, varying only template-generator input; each skill gets at most five templates, generated with GPT-4o-mini at temperature 0.3.Shared controls include preprocessing, output schema, template validation, query instantiation, deduplication, and evaluation metrics.Source: A Pseudo-Query Generation Experiment… · p. 10

A.1 Baseline Methods and Input Construction · A.2 Query Template Generation Prompt · A.3 Parameter Filling and Pseudo-Query Instantiation

The appendices describe controlled comparisons of pseudo-query generation inputs, a shared query-template prompt, and the staged process for filling parameter slots. They contrast metadata, examples, and unstructured-document inputs with Skill2Query’s use of structured skill information and parameter constraints.

A.1 Baseline Methods and Input Construction

  • All generation methods share the same template, parsing, parameter-filling, post-processing, and metric procedures; only their input information and organization differ.Table 8 presents the input-information configurations being compared.Source: §A.1 · p. 10, Table 8 · p. 10
  • Zero-shot LLM uses skill name and description; Few-shot LLM adds up to three usage examples; SkillFlow-style uses the full document unstructured; Skill2Query uses a parsed SKG.The SKG organizes capabilities, parameter schemas, and usage examples for template generation and parameter instantiation.Source: §A.1 · p. 10, Table 8 · p. 10

A.2 Query Template Generation Prompt

  • All approaches reuse the same Query Template Generator, system prompt, output format, and post-processing procedures; Table 9 shows the shared prompt.This setup is intended to ensure a fair comparison among generation methods.Source: §A.2 · p. 10, Table 9 · p. 10
  • Under these shared constraints, the generator produces concise, realistic, structurally valid query templates that follow the provided capability and parameter information.Source: §A.2 · p. 11

A.3 Parameter Filling and Pseudo-Query Instantiation

  • The shared pipeline generates templates with parameter placeholders, fills the slots, then expands pseudo-queries; baseline filling relies on inferred or identified slots and default or fallback strategies.Because these methods lack complete structured parameter schemas, they do not explicitly constrain parameter types, requiredness, defaults, enumerated values, or value ranges.Source: §A.3 · p. 11
  • Skill2Query uses SKG parameter nodes and attributes to guide template generation and filling, selecting concrete values under constraints on types, requiredness, defaults, enumerations, examples, and ranges.It also uses parameter-expression patterns extracted from usage examples.Source: §A.3 · p. 11

A.4 Pseudo-Query Generation Quality Metrics … A.4.3 Distinct-3.

The appendix evaluates pseudo-query quality through parameter validity, functional coverage, and linguistic diversity. Exec-Pass checks parameter constraints, Func-Coverage measures capability coverage, and Distinct-3 captures trigram-level diversity without establishing functional correctness or parameter validity.

A.4 Pseudo-Query Generation Quality Metrics

  • Pseudo-query quality is assessed from three perspectives: parameter validity, functional coverage, and linguistic diversity.Source: §A.4 · p. 11

A.4.1 Exec-Pass.

  • Exec-Pass checks whether instantiated query templates satisfy required-parameter, type, value-range, and enumeration constraints.Source: §A.4.1 · p. 11

A.4.2 Func-Coverage.

  • Func-Coverage measures the extent to which generated queries for a skill cover its capability set.Source: §A.4.2 · p. 11
  • Overall Func-Coverage averages skill-level coverage only across skills with successfully parsed, non-empty capability sets.Source: §A.4.2 · p. 11

A.4.3 Distinct-3.

  • Distinct-3 [16] measures generated-corpus linguistic diversity at the trigram level, with higher scores indicating more varied lexical combinations and expression patterns.Source: §A.4.3 · p. 11
  • Distinct-3 does not directly indicate whether generated queries are functionally correct or parameter-valid.Source: §A.4.3 · p. 11

B Implementation of the online Query Expansion System

The online query expansion system generates two skill-contextualized variants while retaining the original query, then retrieves and fuses results from all three query views. Its prompt constrains rewrites to preserve intent, and generation failures fall back to the original-query retrieval pipeline.

B Implementation of the online Query Expansion System

  • GPT-4o-mini generates two query variants in one call using the original query and SKG-based context from seed skills.The number of generated variants is fixed at two per original query.Source: B Implementation of the online Query… · p. 11
  • The prompt requires intent-preserving, semantically equivalent but varied rewrites and prohibits adding unsupported tasks, tools, interfaces, parameter values, or execution conditions.It aims to normalize colloquial, concise, or incomplete requests to better match the skill retrieval corpus.Source: B Implementation of the online Query… · p. 11
  • The system retains the original query and combines it with the two generated variants, yielding three query views per test instance.Keeping the original prevents generated rewrites from replacing the original query in the expanded set.Source: B Implementation of the online Query… · p. 11
  • Each query view retrieves independently from the same skill index with the same base retriever, returning a top-20 list; unsupervised RRF with k = 60 fuses the lists into the final top 20.If generation fails or produces no valid variants, the system retries up to two times and then returns results from retrieval using only the original query.Source: B Implementation of the online Query… · p. 12

C Downstream Skill Retrieval Training Setup … C.4 Training Configuration

The section describes a controlled SkillRouter fine-tuning pipeline for comparing pseudo-query generation methods, keeping the retrieval models and training setup fixed while varying the generated training data. It specifies how queries and skills are sampled, encoded, and used in embedding and reranking objectives.

C Downstream Skill Retrieval Training Setup

  • Only the pseudo-query generation method varies when fine-tuning SkillRouter-Embedding-0.6B for dense retrieval and SkillRouter-Reranker-0.6B for reranking.Model architectures, input formats, training objectives, data-sampling strategies, and hyperparameters are fixed across experiments.Source: C Downstream Skill Retrieval Training… · p. 12

C.1 Training Data Construction

  • Training sets use at most one template and one sampled pseudo-query per skill, approximately maintaining one query per skill and preventing prolific skills from dominating.All methods use the same sampling strategy and training pipeline so retrieval differences primarily reflect pseudo-query quality.Source: §C.1.1 · p. 12

C.1.2 Two-Stage Training Data Construction

  • Embedding training pairs each pseudo-query with its positive skill, while reranker groups contain one query, one positive, and multiple negative skills.A unique-skill batch sampler avoids treating same-skill instances as in-batch negatives; candidate groups are formed from the top 200 retrieved records, deduplicated and reduced to 10, with the positive replacing the lowest-ranked candidate if absent.Source: §C.1.2 · p. 12

C.2 Model Input Formats; C.2.1 Embedding Model Input

  • The embedding model independently encodes queries and skill documents, using an instruction-prefixed query and a skill representation built from its name, description, and body.Inputs are capped at 1,500 query characters, 300 description characters, and 2,500 body characters, with a 2,048-token maximum for each encoder.Source: §C.2.1 · pp. 12–13

C.2.2 Reranker Model Input

  • The reranker jointly encodes each query with a candidate skill containing its name, description, and body.Its inputs are capped at 1,500 query characters, 500 description characters, and 2,000 body characters, with a 4,096-token maximum.Source: §C.2.2 · p. 13

C.3.1 Embedding Objective

  • The embedding model uses unidirectional query-to-skill in-batch InfoNCE, treating each paired skill as positive and other batch skills as negatives, with temperature 0.05.The objective uses cosine similarity between normalized query and skill embeddings [24].Source: §C.3.1 · p. 13

C.3.2 Reranker Objective

  • The reranker scores query-candidate pairs by the final-position yes-versus-no logit difference and trains with listwise cross-entropy at temperature 1.0.The objective compares candidate relevance scores within each group to promote a higher rank for the unique positive skill.Source: §C.3.2 · p. 13

C.4 Training Configuration

  • All pseudo-query generation methods use the same training hyperparameters; the embedding and reranker fine-tuning configurations are summarized in Table 11.The supplied passage does not provide the table's configuration values.Source: §C.4 · p. 13

D Skill2Query Ablation Study

Ablations show that the Skill Knowledge Graph (SKG) supports semantic grounding and validity, Param-Aware generation primarily supports executability, and Exec-Verify filters invalid outputs with little effect on average metrics.

D Skill2Query Ablation Study

  • Without SKG, Func-Coverage fell from 11.32% to 2.41% and Exec-Pass from 42.85% to 22.63%, showing the most pronounced functional-coverage loss.The SKG links Capability, Parameter, and Example information to guide generation from skill structure rather than plain-text descriptions.Source: D Skill2Query Ablation Study · pp. 13–14, Table 12 · p. 14
  • Without SKG, Distinct-3 rose from 85.71% to 96.74%, reflecting more dispersed wording rather than improved functional correctness or parameter consistency.The increased lexical diversity does not offset the concurrent drops in Exec-Pass and Func-Coverage.Source: D Skill2Query Ablation Study · p. 14
  • Removing Param-Aware reduced Exec-Pass from 42.85% to 21.66%, while Func-Coverage shifted from 11.32% to 11.25% and Distinct-3 from 85.71% to 85.54%.Explicit parameter slots and parameter-type information support downstream parsing and execution even when plain-text templates cover similar capabilities.Source: D Skill2Query Ablation Study · p. 14, Table 12 · p. 14
  • Removing Exec-Verify changed Exec-Pass from 42.85% to 42.90%, Func-Coverage from 11.32% to 11.27%, and Distinct-3 from 85.71% to 85.62%, all by less than 0.1 percentage points.The verifier still filters invalid parameter names, incompatible types, out-of-range values, and illegal enumerations, preventing abnormal results from entering the pseudo-query corpus.Source: D Skill2Query Ablation Study · p. 14, Table 12 · p. 14

Related papers

Loading 2608.16071v1…