Source-linked AI summary
SkillNet: Create, Evaluate, and Connect AI Skills
Yuan Liang, Ruobin Zhong, Haoming Xu, Chen Jiang, Yi Zhong, Runnan Fang, Jia-Chen Gu, Shumin Deng, Yunzhi Yao, Mengru Wang, Shuofei Qiao, Yida Xue, Xin Xu, Tongtong Wu, Kun Wang, Yang Liu, Zhen Bi, Jungang Lou, Yuchen Eleanor Jiang, Hangcheng Zhu, Gang Yu, Haiwen Hong, Longtao Huang, Hui Xue, Chenxi Wang, Yijun Wang, Zifei Shan, Xi Chen, Zhaopeng Tu, Feiyu Xiong, Xin Xie, Peng Zhang, Zhengke Gui, Lei Liang, Jun Zhou, Chiyu Wu, Jin Shang, Yu Gong, Junyu Lin, Changliang Xu, Hongjie Deng, Wen Zhang, Keyan Ding, Qiang Zhang, Fei Huang, Ningyu Zhang, Jeff Z. Pan, Guilin Qi, Haofen Wang, Huajun Chen
TL;DR
AI agents lack systematic mechanisms for consolidating episodic experience into durable, transferable skills. SkillNet addresses this gap by creating, organizing, and evaluating skills from heterogeneous sources through a unified ontology and open infrastructure. Across three simulated environments, SkillNet improves average reward by 40% and reduces interaction steps by 30%.
Problem
Agents lack a unified mechanism for consolidating episodic experience into durable, transferable skills, limiting reliable reuse of prior strategies.
Method
SkillNet creates skills from heterogeneous sources, organizes them through a three-layer ontology, and evaluates them across five quality dimensions.
Results
Across ALFWorld, WebShop, and ScienceWorld, SkillNet improves average reward by 40% and reduces interaction steps by 30%.
Takeaways & Limitations
SkillNet provides an open infrastructure for cumulative agent improvement, reliable cross-task performance, and skill composition.
Takeaways & Limitations
Skill coverage is incomplete, self-constructed skill quality is not fully guaranteed, and adversarial skills cannot be fully mitigated.
Abstract
from arXiv · showhide
Current AI agents can flexibly invoke tools and execute complex tasks, yet their long-term advancement is hindered by the lack of systematic accumulation and transfer of skills. Without a unified mechanism for skill consolidation, agents frequently ``reinvent the wheel'', rediscovering solutions in isolated contexts without leveraging prior strategies. To address this challenge, we introduce SkillNet, an open infrastructure for creating, evaluating, and organizing AI skills at scale. SkillNet structures skills within a unified ontology that supports creating skills from heterogeneous sources, establishing rich relational connections, and performing multi-dimensional evaluation across Safety, Completeness, Executability, Maintainability, and Cost-awareness. Our infrastructure integrates a repository of over 600,000 skills, an interactive platform, and a versatile Python toolkit. Experiments on ALFWorld, WebShop, and ScienceWorld show 40% higher average rewards and 30% fewer execution steps across multiple backbone models. Furthermore, SkillNet-Gym benchmarks skill retrieval, utilization, and composition, while SkillNet-Fabric enables task-specific skill routing through lightweight Wikis. By formalizing skills as evolving, composable assets, SkillNet provides a robust foundation for agents to move from transient experience to durable mastery.
1 Introduction
SkillNet addresses the lack of systematic skill consolidation by turning fragmented experience into durable, composable skills. It combines structured organization, multi-dimensional evaluation, open resources, benchmarks, and task-specific routing to improve agent capability.
- Motivation: Agents struggle to convert episodic experience into durable, transferable mastery and repeatedly rediscover solutions without shared skill consolidation.Existing approaches rely heavily on manual engineering or transient in-context learning.
- Results: 40% higher average reward and 30% fewer interaction steps were achieved across ALFWorld, WebShop, and ScienceWorld with SkillNet augmentation.The improvements were reported across multiple backbone models.
- Contribution: SkillNet organizes skills as modular, composable entities within a structured network that supports actionable knowledge engineering.The framework is designed to transform fragmented experience into reusable capabilities with relational modeling.
- Contribution: Its evaluation protocol measures safety, completeness, executability, maintainability, and cost-awareness for large-scale skill repositories.These dimensions provide intrinsic quality measurements rather than relying only on downstream task success.
- Extensions: SkillNet-Gym evaluates skill construction, retrieval, and composition, while SkillNet-Fabric routes task-specific skill sets through lightweight Wikis.Experiments identify limitations in open-world retrieval and multi-skill orchestration, while routing evaluations show improved quality and task performance.
2 Agent Skills
Agent skills are lightweight, reusable capability packages that provide procedural knowledge and context-specific instructions on demand. They are discovered progressively, activated through full instructions, and executed with optional bundled resources.
- Concept: Skills encapsulate procedural knowledge, task-specific instructions, and supporting resources to extend agent capabilities.They are intended to make complex task performance more accurate, efficient, and consistent.
- Representation: A skill is typically a structured folder centered on SKILL.md, with metadata, execution instructions, and optional scripts, templates, or documentation.The package forms a self-contained capability for agent use.
- Use: Skills provide reusable procedural knowledge and context-specific information for tasks such as data analysis, domain reasoning, and structured output generation.Their purpose is on-demand access rather than permanent inclusion in the agent’s context.
- Benefits: Skills reduce reliance on hard-coded rules or ad hoc prompts by enabling consistent, repeatable workflows and knowledge sharing across agents and applications.They also support iterative auditing and improvement through embedded instructions and metadata.
- Workflow: The progressive workflow is discovery, activation, and execution: agents first inspect metadata, then load instructions and resources, and finally follow them.Bundled code or referenced assets may be used during execution.
3 SkillNet
SkillNet creates, evaluates, analyzes, and distributes reusable skills from heterogeneous sources through a layered ontology and open infrastructure. Its curation and evaluation processes support large-scale repositories while exposing executability as a persistent quality challenge.
- 3.1 Overview: SkillNet transforms user trajectories, documents, repositories, prompts, and internet resources into executable, reusable skills.Its automated creation pipeline extracts executable patterns and applies multi-stage filtering, deduplication, categorization, evaluation, and consolidation.
- 3.2 Skill Ontology: The Skill Ontology combines functional taxonomy, inter-skill relations, and modular skill packages for categorization, reasoning, composition, and deployment.Relations include similarity, composition, membership, and dependency, while the ontology can evolve through added tags and inferred relations.
- 3.1 Overview: SkillNet’s core modules create skills, evaluate them across safety, completeness, executability, maintainability, and cost-awareness, and analyze structural relationships.The resulting graphs support retrieval, composition, and workflow synthesis.
- 3.4 Skill Evaluation: MAE below 0.03 and QWK above 0.90 across dimensions show close agreement between automated evaluation and human reviews of 200 sampled skills.The repository still contains a larger fraction of Average ratings for Executability than for Safety and Maintainability.
- 3.5 Open Resources: More than 600k candidate skills are filtered into a repository of over 500k curated skills, which is continuously expanding.The repository is paired with a website, open API, Python toolkit, and task-specific collections.
4 Quantitative Evaluation
Across ALFWorld, WebShop, and ScienceWorld, SkillNet improves task effectiveness and execution efficiency across diverse backbone models. The gains remain robust across model capacities and seen versus unseen settings.
- Quantitative Results: SkillNet consistently outperforms ReAct and Few-shot baselines in average rewards while reducing average steps across ALFWorld, WebShop, and ScienceWorld.Figure 6 reports rewards in the top panels and steps in the bottom panels.
- Quantitative Results: 40% higher average reward and 30% fewer interaction steps are achieved with SkillNet than ReAct on average across the evaluated environments and models.The results indicate more reliable task solving with shorter action trajectories.
- Robustness: +15.7 R for o4 Mini and +28.5 R for Gemini 2.5 Pro show robust improvements across compact and large-scale backbone models.The gains persist across backbone models of varying capacities.
- Robustness: SkillNet gains remain consistent in both seen and unseen settings, supporting knowledge transfer across tasks and environments.The reported pattern is associated with skill abstraction and reuse.
- Implications: Formalizing skills as independently accumulated, knowledge-grounded capability units supports cumulative rather than episodic enhancement of agent competence.The paper connects this design principle with domain-specialized and continually self-improving agents.
5 SkillNet-Gym
SkillNet-Gym is constructed from real-world skill ecosystems to evaluate skill construction, retrieval, and composition across diverse task structures. Its results expose stronger benefits on hard and compositional tasks, while also revealing limitations in one-shot construction and open-world orchestration.
- Benchmark Design: SkillNet-Gym is grounded in real community resources and spans 13 core domains and 81 subdomains.The benchmark is designed to reflect the variability and messiness of real-world usage scenarios.
- Benchmark Construction: SkillNet-Gym synthesizes tasks by sampling chains, fan-out, fan-in, and diamond subgraphs from a directed skill graph.Each candidate preserves participating skills, directed connections, and scenario-level evidence for graph edges.
- Benchmark Construction: 159 high-quality tasks remain after multi-stage quality control covering skill graphs, instructions, reference solutions, test scripts, and task difficulty.The procedures include filtering low-quality nodes and edges and task-level checks.
- Main Results: Hard tasks generally receive larger and more consistent relative gains from skill augmentation than easy tasks.The paper attributes this pattern to limited improvement headroom on easy tasks for capable frontier models.
- Limitations: One-shot skill construction can discard information from dense documents and degrade performance on tasks requiring specific capabilities.The study evaluates only a single-pass compression strategy and notes that curated skills are not always superior to self-constructed ones.
- Main Results: Large community-sourced skill repositories can reduce performance because incomplete retrieval and incorrect orchestration directly limit downstream task success.The degradation is more pronounced on hard tasks than when official skills are directly provided.
- Main Results: Specialized composition baselines outperform direct agent orchestration, with AgentSkillOS substantially exceeding repository-based orchestration.The results support hybrid retrieval and orchestration strategies for robust skill composition in the wild.
- Analysis: Skill utility varies by domain, with gains of +50.0 pp in Geospatial Analysis and +33.3 pp in Information Extraction but declines of -6.2 pp in Academic Research and -4.2 pp in Business Compliance.Procedural workflows benefit most, whereas analytics and visualization show moderate gains and substantial reasoning remains with the model.
6 SkillNet-Fabric
SkillNet-Fabric routes task-relevant skills from a large ecosystem through a task-specific Wiki, separating candidate availability from final skill-set formation. Across routing and downstream benchmarks, it improves both complete-set recovery and task performance, while results show that broader candidate spaces can increase effort without improving selection.
- Framework: SkillNet-Fabric places a task-specific Wiki between the global skill ecosystem and final set formation, giving the Explorer retained candidates and supporting evidence.The routing layer combines retrieval and relation expansion before the Explorer forms the final skill set.
- Routing task: The routing task selects up to K skills whose combined capabilities and instructions should cover a task’s requirements.Candidate skills and evidence define the routing space from which the final set is formed.
- Build-time representation: SkillNet-Fabric converts heterogeneous sources into contracts, typed relations, retrieval indexes, and a global Wiki for reusable, source-grounded skill representation.Contracts retain capability, conditions, inputs, and outputs linked to supporting source lines; typed relations encode dependency, composition, and similarity.
- Routing quality: SkillNet-Fabric achieves 67.91% Recall and 49.33% FullCoverage at K=5, rising to 74.80% and 60.00% at K=10 on SkillRouter Hard.At K=10, these results exceed the official SkillRouter method by 5.38 and 9.33 percentage points, respectively.
- Downstream performance: SkillNet-Fabric achieves the highest overall verifier reward in both SkillsBench model conditions and ranks first in all six AgentSkillOS model–pool combinations.On SkillsBench, rewards are 46.08% with GPT-5.4 mini and 66.67% with GPT-5.6 Terra; AgentSkillOS Pool Mean scores are 64.21 and 83.00.
- Candidate scope: Candidate availability improves as L grows, but final complete-set recovery depends on Explorer-specific scope and can decline when the candidate space becomes too broad.The full pool can increase token use without improving Final FullCoverage@10, supporting a two-stage routing design.
7 Application Scenarios
SkillNet application scenarios organize specialized skills into executable workflows for scientific discovery and software engineering. The OpenClaw integration extends this pattern with dynamic acquisition, quality-aware management, and experience-driven skill creation.
- Scientific discovery: The scientific-discovery scenario schedules data processing, mechanistic analysis, target validation, and report-generation skills into a research workflow.The workflow processes single-cell RNA-seq data, maps genes to pathways, validates clinical significance, and produces cited reports.
- Scientific discovery: SkillNet’s scientific workflow transforms fragmented research skills into a structured, composable network for an AI Scientist.The scenario is presented as an illustrative prototype rather than a real-world application.
- Coding: The coding scenario coordinates code analysis, requirements decomposition, impact analysis, implementation, testing, and validation for production-grade software evolution.It targets correctness, performance, maintainability, and regression-risk assessment within an existing codebase.
- Coding: The coding scenario frames complex software engineering as capability composition and coordination across coding, analysis, and testing skills.This organization is presented as a scalable automation paradigm for system-level software evolution.
- OpenClaw integration: In OpenClaw, SkillNet supports dynamic skill acquisition, quality-aware library management, and experience-driven knowledge creation through lazy-loaded skills.The integration searches for skills before unfamiliar tasks, downloads matched skills, and reads SKILL.md instructions on demand.
- OpenClaw integration: The OpenClaw–SkillNet integration forms a closed loop in which community skills inform execution, successful outcomes become new skills, and periodic analysis maintains repository quality.This circulation supports the paper’s vision of scalable, evolving agent intelligence.
8 Related Work
Related platforms largely distribute skills as static packages or marketplace assets, with limited automated creation and multidimensional evaluation. SkillNet positions itself as a full-lifecycle infrastructure combining automated creation, rigorous evaluation, and relational organization.
- Agent skill research: Prior work explores failure summarization, corrective feedback, long-term memory, and structured skill acquisition for agent systems.These directions address learning from interaction and experience but are presented as distinct strands of related research.
- Skill platforms: Community skill repositories and evaluations promote reuse, standardization, and ecosystem collaboration through directories, package hubs, and marketplaces.Examples include ClawHub, SkillsMP, Skills.sh, and SkillHub.
- Limitations: Existing platforms rely largely on manual curation and ad hoc quality control, while evaluation emphasizes community metrics or end-task performance.The comparison identifies these practices as limitations of distribution-centric skill platforms.
- SkillNet’s position: SkillNet addresses these bottlenecks with automated skill creation, multidimensional evaluation, and relational connectivity through a structured Skill Graph.The framework is presented as full-lifecycle infrastructure for skill organization, verification, and evolution.
9 Conclusion, Discussion and Future Work
SkillNet frames skills as reusable units that connect workflows, memory, and generative capability, enabling cumulative improvement and collaboration while leaving open-world evolution and model–skill synergy unresolved.
- Conclusion: SkillNet consolidates experience into structured, executable skills to support cumulative improvement, reliable task performance, and robust composition.The framework is presented as a foundation for scalable continual learning and evolving intelligence.
- Discussion: SkillNet packages reusable capability units that bridge rigid workflows and adaptive memory.Skills constrain generation while organizing accumulated experience.
- Future Work: SkillNet makes skills a shared layer for collaborative planning, knowledge transfer, and experience accumulation across agents.Individuals can curate repositories while agents compose skills into workflows and memory refines them through experience.
- Future Work: Automatic skill discovery, abstraction, and cross-domain transfer remain challenging in open-world settings requiring dynamic composition of complex tasks.Industrial manufacturing, finance, and scientific research are identified as demanding settings.
- Future Work: The synergy between executable skills and underlying model capabilities remains largely unexplored, including neuro-symbolic integration and evolving skill hierarchies.The paper identifies dynamic restructuring of skill dependencies as an open research question.
10 Limitations
The paper compares skill benchmarks across construction, scope, and evaluation traits, while recognizing that present SkillNet coverage and quality assessment remain incomplete.
- Limitations: SkillNet coverage is incomplete, self-constructed skill quality is not fully guaranteed, poisoned contributions are not fully mitigated, and an end-to-end agent-instantiation pipeline is absent.These limitations constrain repository completeness, assurance, safety, and automation scope.
- Benchmark Scope: Benchmark comparisons span construction, scope, and evaluation focus across eight traits.The traits include execution, real-world sourcing, automatic construction, extensibility, domain coverage, skill pool, task numbers, and skill construction or composition.
- Benchmark Construction: Real-world sourcing distinguishes artifacts, repositories, or tasks from community environments from primarily synthetic or hand-crafted benchmarks.This trait assesses grounding in realistic sources rather than evaluation convenience.
- Benchmark Scope: Dynamic extensibility measures whether benchmarks can be updated as new skills, artifacts, or workflows emerge.It distinguishes evolving benchmarks from static snapshots.
- Evaluation Focus: Skill composition evaluates whether agents retrieve, organize, or combine multiple skills into coherent workflows.This differs from treating skills as isolated, single-use aids.
- Evaluation Focus: Skill execution measures end-to-end solving with executable environments and deterministic verification.It focuses on whether skill use improves real execution outcomes.
B Further Experiment Details
Further experiments show that skill construction methods can complement one another across models, while large open-world libraries expose retrieval and orchestration weaknesses.
- Skill Construction: Different skill construction methods can play complementary roles across models.The comparison is reported in Table 9 under the skill construction setting.
- Skill Construction: Documentation compression limits both evaluated skill construction methods.The paper points toward methods beyond static distillation.
- Wild Setting: Over 2000 skills create a wild-setting evaluation of retrieval and task performance.Agents are tested with a large skill library rather than only task-provided official skills.
- Wild Setting: Performance drops on Easy and Hard tasks relative to providing the correct official skills, with degradation particularly pronounced on Hard tasks.The paper attributes the decline to retrieving relevant skills or only an incomplete subset.
- Wild Setting: Effective skill usage in realistic open-world scenarios requires a strong skill composition baseline.The results link the observed degradation to failures in retrieval and incomplete skill subsets.
C.1 Benchmark Construction Details
SkillNet-Gym is constructed through a staged pipeline that retrieves and cleans skills, extracts scenarios, verifies compositional edges, and quality-controls synthesized benchmark artifacts.
- Graph Construction: SkillNet construction turns community skill retrieval results into a scenario-mediated directed skill graph.The pipeline supports downstream task synthesis through verified compositional relationships.
- Skill Retrieval: Skill retrieval uses diverse query seeds, vector search, similarity thresholds, bounded top-k retrieval, ranking, deduplication, and GitHub materialization.A manifest records provenance and download status for the resulting local corpus.
- Skill Cleaning: Near-duplicate skills are embedded and clustered with FAISS and union-find, retaining the highest-star representative.This reduces redundancy in the candidate skill corpus.
- Scenario Modeling: Each skill is converted into pre- and post-scenarios, which are normalized, embedded, and clustered before alignment.The scenarios represent states before applicability and after successful execution.
- Edge Verification: Compatible scenario pairs are retrieved and then judged for whether one skill’s post-state naturally satisfies another skill’s pre-state.Verified alignments become directed compose_with edges containing scenario-level evidence and confidence information.
- Quality Control: Quality control removes redundant skills and edges while checking skill cost, verifiability, documentation, and artifact relevance.Task-level control also covers instructions, reference solutions, test scripts, and difficulty.
- Task Validation: Synthesized tasks are filtered for difficulty and cross-validated to exclude shortcut-prone tasks solvable without skills.Additional judges and post-hoc verification assess instructions, solutions, and tests.