Source-linked AI summary
Code Skill Server: A Large-Scale Search Engine for Code Generation Skills
Xiaoyong Li, Haolong Li, Dongliang Xu, Haoyu Li, Yifan Wang, Jingyuan Shi, Yanghua Wang, Yanbin Sun, Shidong Pan, Yue Yao
TL;DR
Finding appropriate skills across heterogeneous sources is difficult to do manually at scale, while existing retrieval methods mainly focus on established libraries. CSS collects and indexes public code-generation skills and serves task-relevant skills to client agents; its best configurations improve benchmark pass rates, though task-level outcomes vary.
Problem
Manual discovery and organization of skills across heterogeneous sources is increasingly impractical, while existing retrieval methods mainly focus on skills in established libraries.
Source: §1 · p. 2Method
CSS collects, standardizes, and organizes public code-generation skills, then serves retrieved skills through a shared interface to client agents.
Source: §1 · p. 2Results
Best-performing CSS configurations raise task pass rates from 53.42% to 60.88% on BigCodeBench and from 33.04% to 46.96% on CoderEval.
Source: §1 · p. 2Takeaways & Limitations
CSS is effective across different code-generation settings and models, and corpus scale and construction influence downstream performance.
Source: §5 · p. 9Skill augmentation is task-dependent and does not guarantee a positive change on every task.
Source: §4.3 · p. 8
Abstract
from arXiv · showhide
Agent skills have proven to be an effective technique for improving agent ability on specific tasks, including software engineering, natural science, and mathematics. During such a process, using agent skills often involves manually discovering and organizing skills from heterogeneous sources, while existing skill retrieval methods mainly focus on selecting relevant skills from established skill libraries. However, with the rapidly increasing number and variety of reusable agent skills, manually searching for relevant skills across heterogeneous sources is becoming impractical at scale. In this paper, we introduce the Code Skill Server (CSS), a large-scale search engine for finding the most useful code generation skills for a target task. CSS consists of a skill server that incrementally collects and indexes publicly available code generation skills from multiple sources, with 993 skills included in the corpus used in this work. Extensive experiments across different retrieval methods and code generation benchmarks show that the best CSS configuration improves Pass@1 by up to 13.91% and consistently improves all evaluated models by at least 4.21% in the cross-model evaluation. These results demonstrate that CSS can effectively search for and serve relevant skills, improving code generation quality and enhancing models' code generation capabilities. Code and the CSS-993 corpus are available at {https://github.com/haoyuli-sdu/CodeSkillServer}.
1 INTRODUCTION
CSS addresses the difficulty of finding applicable skills across heterogeneous sources by integrating skill collection, organization, retrieval, and agent use. Its evaluation compares retrieval configurations on two benchmarks and across models.
1 INTRODUCTION
- Practical code generation remains challenging because models must handle complex requirements, external libraries, and dependencies in surrounding projects.Reusable skills can provide domain knowledge and procedural guidance alongside model knowledge.Source: §1 · p. 1
- Manual skill selection spans multiple sources and requires inspecting applicability, while existing retrieval methods mainly address selection and contextualization within libraries.The introduction identifies integration of external skill collection with retrieval and subsequent agent use as a remaining need.Source: §1 · pp. 1–2
- CSS collects publicly available skills, standardizes and organizes them, then retrieves task-relevant skills for client agents from a shared collection.The initial collection contains 993 code generation skills; the server selects skills to serve, while the agent decides how to use them.Source: §1 · p. 2
- Across BigCodeBench, best-performing CSS configurations raise task pass rates from 53.42% to 60.88%, while on CoderEval they raise rates from 33.04% to 46.96%, over three retrieval configurations.The evaluation spans five models.Source: §1 · p. 2
2 RELATED WORK
Related work covers reusable agent skills, methods for organizing and retrieving them, and code-generation benchmarks. CSS focuses on externally collected code skills and evaluates their effect on functional correctness.
Agent Skills
- Prior skill work spans reusable experience and workflows, automatic skill discovery and evolution, dependency-aware retrieval, and systematic evaluation of skill effectiveness.CSS complements these studies by organizing externally collected code skills into a shared resource and providing a common interface for comparing retrieval methods.Source: §2 · pp. 2–3
Code Generation
- Code-generation benchmarks range from function-level unit-test evaluation to realistic library and project settings, including BigCodeBench and CoderEval.The paper uses these benchmarks to assess whether retrieving externally sourced skills improves generated-code functional correctness.Source: §2 · p. 3
3 CODE SKILL SERVER
CSS pairs a SkillServer that indexes skills with a client agent that uses retrieved skills for a coding task. The collection is normalized and organized to support task-specific retrieval.
3 CODE SKILL SERVER
- For each coding task, the client submits a search request, the server returns skills using its configured retrieval method, and the client agent decides how to use them.The SkillServer maintains an indexed skill collection, while clients are equipped with code generation agents.Source: §3 · p. 3
- CSS filters and normalizes heterogeneous skills into structured records and capability pools, then indexes them for retrieval.The task query includes retrieval constraints; the server returns ordered skills and the agent chooses which to use.Source: §3 · p. 3
3.1 PROBLEM DEFINITION
The problem is to select skills relevant to a target coding task from a potentially large, heterogeneous collection. CSS organizes and searches that collection so clients need not inspect it manually for each task.
3.1 PROBLEM DEFINITION
- A task combines a natural-language specification with available coding context, while each skill gives reusable guidance for a class of tasks rather than a specific solution.The context can include programming language, function signature, library dependencies, and input and output requirements.Source: §3.1 · p. 3
- The selection goal is to choose a subset Sq of the indexed skill collection S that is relevant to target task q.The ideal selection is framed in terms of the relevance of a candidate skill subset to the task.Source: §3.1 · p. 3
- Large heterogeneous collections make selection challenging because skills differ in structure, metadata, naming, and capability descriptions.CSS searches and organizes skills on the server and retrieves relevant ones for each client task.Source: §3.1 · p. 3
3.2 SERVER DESIGN
The SkillServer prepares public skills independently of client requests, turning heterogeneous documents into searchable records and organizing them into capability pools. These pools support retrieval over relevant categories while retaining access to complementary skills.
3.2.1 SKILL SEARCH AND ORGANIZATION
- CSS discovers public skills, filters for code-producing instructions, normalizes retained documents, and removes exact duplicates while preserving source provenance.Skills with the same name but different contents remain distinct.Source: §3.2.1 · p. 4
- Prepared skill records support content-based and relation-based retrievers through reusable indexes and a common retrieval interface.The indexes are reused across requests rather than rebuilt for each one.Source: §3.2.1 · p. 4
3.2.2 SKILL CAPABILITY POOLS
- Each skill record stores its name, description, instructions, primary capability category, and provenance.The seven categories organize skills by their main code artifact or development activity.Source: §3.2.2 · p. 5
- Capability pools are logical views of one shared collection, and a task may search the union of multiple pools.This organization lets the server restrict matching to a relevant candidate space while retaining complementary capabilities.Source: §3.2.2 · p. 5
3.3 SKILLSERVER-CLIENT TRANSACTIONS
CSS transactions connect task adaptation, server-side skill selection, and client-side code generation. FASTADAPT constructs a structured request, the server returns relevant skills, and the client decides how to use them.
3.3 SKILLSERVER-CLIENT TRANSACTIONS
- A CSS transaction has the client construct a task request, the SkillServer select admissible skills, and the client decide which returned skills to use for code generation.The interaction is organized as three steps.Source: §3.3 · p. 5
3.3.1 FASTADAPT
- FASTADAPT uses the client model to form a query and set capability categories, maximum skill count, token budget, and relevance threshold.The retrieval method is configured separately; unspecified capability categories default to the full collection, and unspecified constraints impose no additional limits.Source: §3.3.1 · p. 5
3.3.2 SKILL SELECTION
- The server applies the configured retrieval method to the normalized query and candidate collection, returning selected skills in descending relevance order and potentially fewer than K.The candidate collection can combine skills from multiple capability pools.Source: §3.3.2 · pp. 5–6
3.3.3 CLIENT WITH SKILLS
- The client combines the original task and coding context with served skill instructions, while its agent chooses whether to use those skills individually, together, or not at all.CSS determines which resources are available; the client determines their use during code generation.Source: §3.3.3 · p. 6
4 EXPERIMENTS
The experiments evaluate CSS on function-generation and project-context benchmarks using a shared skill corpus and several retrieval settings. Skills improve benchmark performance, with results also examined by task, model, and corpus characteristics.
4.1 EXPERIMENTAL SETUP
- Evaluation uses BigCodeBench and CoderEval-Python, a shared 993-skill corpus, No-Skill, BM25, Graph-of-Skills, and SkillTrace, with Pass@1 on matched inputs.The main model is GPT-5.6; cross-model evaluation uses four additional models with SkillTrace.Source: §4.1 · pp. 6–7
4.2 EFFECTIVENESS ACROSS CODE GENERATION SETTINGS
- 60.88% Pass@1 on BigCodeBench versus 53.42% without skills, and 46.96% on CoderEval-Python versus 33.04% without skills.These are the strongest skill-augmented results reported for the two benchmarks.Source: Table 1 · p. 7, §4.2 · p. 7
- SkillTrace improves Pass@1 in all seven BigCodeBench domains, with Network rising from 40.91% to 62.12%; CoderEval gains span dependency levels, though gains vary by setting.The Network increase is 21.21%.Source: §4.2 · p. 7
4.3 TASK-LEVEL IMPACT ANALYSIS
- On BigCodeBench, 112 failed tasks become successful and 27 successful tasks regress, for a net increase of 85 successes.SkillTrace recovers 21.1% of tasks failing without skills, while 4.4% of tasks that originally pass regress; transitions also vary by domain.Source: §4.3 · p. 8, Figure 4 · p. 8
4.4 CROSS-MODEL GENERALIZATION
- All five evaluated models improve with SkillTrace, by 4.21 to 7.46 percentage points; gain size is not monotonic with No-Skill performance.GLM-5.3-Flash has the highest No-Skill result but the smallest gain, while GPT-5.6 and DeepSeek-V4-Flash each gain more than seven percentage points.Source: Figure 5 · p. 8, §4.4 · p. 8
- Cross-model domain profiles differ: General and Cryptography are stronger domains for most models, while Time remains challenging for several.Table 2 reports SkillTrace Pass@1 across seven BigCodeBench domains and overall for five models.Source: §4.4 · p. 9, Table 2 · p. 9
4.5 ANALYSIS OF THE CSS SKILL CORPUS
- Pass@1 rises from 48.10% without skills to 51.90%, 56.67%, 57.14%, and 58.10% as the corpus grows from 20% to 100%.The comparison uses the same 210 BigCodeBench tasks and BM25 throughout.Source: Table 3 · p. 9, §4.5 · p. 9
- With task set, model, BM25 retriever, and prompt structure held constant, CSS-993 reaches 58.10% Pass@1 versus 50.95% for Raw-993.Both corpora contain 993 skills; CSS-993 is collected and organized through the CSS pipeline, while Raw-993 is randomly sampled.Source: §4.5 · p. 9
5 CONCLUSION
The paper presents CSS as a unified service for retrieving reusable skills for code-generation tasks. Experiments show performance across settings and models, with corpus scale and construction also influencing downstream performance.
5 CONCLUSION
- CSS performs across code-generation settings and models, while corpus scale and construction influence downstream performance.The authors identify improving skill selection and supporting larger, more diverse skill ecosystems as future work.Source: §5 · p. 9
- CSS collects public skills, standardizes and organizes them by capability, then serves retrieved task-specific guidance to client agents.The conclusion describes a common retrieval and serving interface for this process.Source: §5 · p. 9
- The paper provides descriptions of its collection, organization, retrieval, model, benchmark, and evaluation procedures to facilitate reproduction.Source: REPRODUCIBILITY STATEMENT · p. 10