Source-linked AI summary
SKILLFOUNDRY: Building Self-Evolving Agent Skill Libraries from Heterogeneous Scientific Resources
Shuaike Shen, Wenduo Cheng, Mingqian Ma, Alistair Turcan, Martin Jinye Zhang, Jian Ma
TL;DR
Scientific procedural knowledge is fragmented across heterogeneous resources, limiting agents’ access to reusable domain expertise. SKILLFOUNDRY builds and validates executable skills through a tree-guided closed loop, producing novel skills that improve benchmark and domain-specific scientific task performance.
Problem
Scientific procedural knowledge is fragmented across repositories, APIs, scripts, notebooks, documentation, databases, and papers, while agents lack the contextual expertise needed for reliable scientific execution.
Method
SKILLFOUNDRY mines heterogeneous resources, extracts operational contracts, compiles executable skill packages, validates them, and iteratively expands, repairs, merges, or prunes a tree-organized library.
Results
SKILLFOUNDRY produces a novel, internally valid skill library and improves agent performance across scientific benchmarks and real-world workflows.
Takeaways & Limitations
Automatically mined skills expand coverage beyond existing libraries and provide reusable capabilities for benchmarked and domain-specific scientific tasks.
Takeaways & Limitations
Current coverage spans only a subset of scientific domains and task families, while most mined skills rely primarily on internal testing and evaluation remains limited to a small number of domains and tasks.
Abstract
from arXiv · showhide
Modern scientific ecosystems are rich in procedural knowledge across repositories, APIs, scripts, notebooks, documentation, databases, and papers, yet much of this knowledge remains fragmented across heterogeneous artifacts that agents cannot readily operationalize. This gap between abundant scientific know-how and usable agent capabilities is a key bottleneck for building effective scientific agents. We present SkillFoundry, a self-evolving framework that converts such resources into validated agent skills, reusable packages that encode task scope, inputs and outputs, execution steps, environment assumptions, provenance, and tests. SkillFoundry organizes a target domain as a domain knowledge tree, mines resources from high-value branches, extracts operational contracts, compiles them into executable skill packages, and then iteratively expands, repairs, merges, or prunes the resulting library through a closed-loop validation process. SkillFoundry produces a substantially novel and internally valid skill library, with 71.1\% of mined skills differing from existing skill libraries such as SkillHub and SkillSMP. We demonstrate that these mined skills improve coding agent performance on five of the six MoSciBench datasets. We further show that SkillFoundry can design new task-specific skills on demand for concrete scientific objectives, and that the resulting skills substantially improve performance on two challenging genomics tasks: cell type annotation and the scDRS workflow. Together, these results show that automatically mined skills improve agent performance on benchmarks and domain-specific tasks, expand coverage beyond hand-crafted skill libraries, and provide a practical foundation for more capable scientific agents.
1 Introduction
SKILLFOUNDRY addresses the gap between fragmented scientific procedural knowledge and agents’ need for reliable, reusable domain expertise. It converts heterogeneous resources into validated executable skills and evaluates their library quality and downstream utility.
- Motivation: Scientific procedural knowledge is fragmented across resources and often unavailable in forms agents can directly use.Reliable scientific execution also depends on specialized conventions, environmental constraints, implicit assumptions, and multi-step workflows.
- Approach: SKILLFOUNDRY converts heterogeneous domain resources into structured, executable, and validated agent skills.The framework extracts operational contracts and iteratively refines the resulting skill library.
- Evaluation: The evaluation combines library-level validation with task-level utility on benchmarks and realistic scientific workflows.Library quality includes executability and novelty, while task-level evaluation tests performance improvements using existing and newly constructed skills.
- Results: SKILLFOUNDRY skills extend beyond existing libraries and improve agent performance on scientific benchmarks.The paper presents this as a central contribution of the framework.
- Results: The framework can design task-specific skills for real-world scientific workflows.This extends the library-building setup to concrete scientific objectives specified for particular tasks.
2 Related Work
Related work spans tool access, reusable skill libraries, automatic tool discovery, and scientific agents. SKILLFOUNDRY is positioned within these efforts as a framework for building skills from heterogeneous scientific resources rather than assuming capabilities already exist in usable form.
- Agent Tools and MCPs: Tool-use research and MCP improve agents’ access to external capabilities, but generally assume those capabilities are already available in usable form.MCP standardizes exposure of tools, resources, and prompts to agents.
- Agent Skills: Prior systems demonstrate reusable executable skill libraries and modular skill packages for agent use.Examples include Voyager’s growing library and Claude’s packages combining instructions, metadata, and optional scripts or resources.
- Tool Ecosystems and Automatic Tool Discovery: Tool ecosystems and automatic discovery convert or expose large collections of scientific capabilities for agents.ToolUniverse standardizes access to more than 1,000 models, datasets, APIs, and scientific packages, while Deploy-Master targets more than 50,000 runnable tools.
- Scientific Agents and Tool Use: Scientific agent systems integrate specialized tools and workflows across domains including chemistry, spatial biology, CRISPR, and broader biomedical science.The cited systems include ChemCrow, SpatialAgent, CRISPR-GPT, and Biomni.
- SKILLFOUNDRY: SKILLFOUNDRY adds a closed-loop process that mines resources, validates skills, expands coverage, and prunes redundant or low-value leaves.Its overview begins with a domain knowledge tree and uses verified skills to update that tree.
3 Method
SKILLFOUNDRY uses a domain knowledge tree to guide resource mining, compile operational contracts into skill packages, validate execution, and reorganize the library from evaluation signals.
- Method Overview: The domain knowledge tree acts both as a search prior and as the object updated by the skill-acquisition loop.Under-covered branches trigger targeted mining, while evaluation signals determine expansion, revision, or pruning.
- Method Overview: The framework selects a branch, mines relevant resources, compiles structured skill cards, tests skills, and refreshes the tree.Skill cards record scope, dependencies, inputs, outputs, sources, and examples before multi-level testing.
- Domain Knowledge Tree: The rooted tree represents domains and subdomains internally, while leaves denote actionable skill targets.It is initialized from an extensible taxonomy and updated as the library grows through splitting, merging, and pruning.
- Resource Mining: Resource search prioritizes branches with abundant resources but weak verified skill coverage rather than searching uniformly.The staged loop is instantiated as tree check → resource search → skill build → skill test → refresh.
- Skill Testing and Tree Refinement: Candidate skills enter the library only after execution, system, and synthetic-data testing establish minimum executability standards.These tests check declared contracts, infrastructure-dependent use, required interfaces, controlled-input behavior, and output stability.
- Skill Testing and Tree Refinement: Evaluation results expand coverage with passing skills and revise, merge, or prune skills that fail repeatedly, duplicate functionality, or add little value.This refinement step closes the loop instead of allowing the library to accumulate skills without reorganization.
4 Experiments
The experiments evaluate SKILLFOUNDRY from complementary perspectives: the quality of its mined skill library and the downstream utility of those skills on scientific tasks.
- Experimental Design: The experimental program characterizes the mined library through statistical analysis and internal validation of validity, novelty, and composition.These analyses assess properties of the library itself before downstream task evaluation.
- Experimental Design: The experiments also test whether existing mined skills improve agent performance on MoSciBench scientific discovery tasks.This measures downstream utility rather than only internal library properties.
286 SKILLS
SKILLFOUNDRY builds and validates a broad, substantially novel scientific skill library, then evaluates whether those skills improve benchmark and domain-specific workflows. The library contains 286 skills, while downstream results show gains across MoSciBench and challenging genomics tasks.
- Library composition and validation: 286 skills span 27 domains and 254 subdomains, mined from 394 resources and retained only after required validation tests.Validation includes execution and synthetic-data testing, with system testing added when cluster infrastructure is required.
- Library composition and validation: 71.1% of mined skills meet the paper’s novelty criterion relative to existing libraries such as SkillHub and SkillSMP.A skill is considered novel when no existing skill is found that completes the same task.
- Mining runtime: Skill extraction is the dominant runtime cost, followed by resource mining, while testing and tree updates add relatively little overhead under a fixed search budget.The reported cost measures runtime for controlled retrieval rather than API token usage or open-ended search.
- Downstream benchmark evaluation: Adding SKILLFOUNDRY skills improves performance on five of six MoSciBench datasets, raising Repo-Acc from 61.19% to 66.73% and Paper-Acc from 43.85% to 53.05%.Code execution success remains 100.00% across all datasets and both settings.
- Task-specific genomics workflows: For spatial-transcriptomics cell annotation, Codex+SKILLFOUNDRY reaches 99.2% coverage and 82.9% accuracy versus 81.1% and 68.5% for vanilla Codex.SpatialAgent performs higher overall with 100.0% coverage and 87.1% accuracy while using an external reference dataset.
- Task-specific genomics workflows: On scDRS, skill-augmented Biomni lowers mean RMSE from 0.11 to 0.02, with two of three runs exactly matching expert outputs.The skill-augmented setting also produces the only run satisfying all qualitative criteria and richer scientific figures.
5 Conclusion
SKILLFOUNDRY converts fragmented scientific procedural knowledge into reusable agent skills through a tree-guided closed loop for mining, validation, and refinement. The resulting library is novel and internally valid, and the skills improve agents on benchmarks and scientific workflows.
- SKILLFOUNDRY converts heterogeneous resources into structured agent skills through tree-guided mining, validation, and refinement.The framework targets repositories, APIs, scripts, notebooks, documentation, databases, and papers.
- The resulting skill library is substantially novel and internally valid.
- SKILLFOUNDRY improves agent performance on standardized benchmarks and real-world scientific workflows.
- The framework provides a path for transforming fragmented scientific know-how into reusable capabilities for more capable, scalable, and reliable scientific agents.
A.1.1 Tree State and Branch Prioritization
SKILLFOUNDRY uses the domain knowledge tree as both a stateful repository index and a controller for prioritizing future skill acquisition. It focuses mining on branches with high expected value and updates the taxonomy according to coverage and validation outcomes.
- Tree State: Each tree node stores linked resources, linked skills, verification outcomes, and coverage state through machine-readable registries.
- Branch Prioritization: SKILLFOUNDRY prioritizes branches with high expected marginal value under the current repository state.Examples include resource-rich leaves without executable skills, low-confidence skill coverage, or repeated validation failures.
- Branch Prioritization: Branch-first prioritization reduces redundant search by turning heterogeneous resource exploration into bounded skill acquisition problems.
- Tree Evolution: The initial taxonomy is broad and manually curated, then revised by splitting, merging, collapsing, or deprioritizing branches based on observed coverage and skill outcomes.
A.1.2 Skill Packaging and Repository Integration
SKILLFOUNDRY retrieves branch-conditioned resources, extracts operational contracts, and compiles them into reusable skill packages. The same contract and testing pathway also supports task-specific skill design.
- Contract Extraction: Branch-conditioned retrieval supplies candidate resources for extracting each skill’s operational contract.Contracts specify scope, inputs, outputs, assumptions, steps, provenance, guidance, and available invocation or test examples.
- Skill Packaging: Compiled skills combine human-readable specifications, machine-readable metadata, provenance, executable scripts, example assets, and repository tests when available.
- Repository Integration: Smoke targets or test commands are paired with candidate skills whenever possible, and their verification status is recorded in a shared registry.
- Task-Specific Design: Task-specific skill synthesis reuses the same contract and packaging pathway rather than creating a separate ad hoc tool-building process.
A.1.3 Hierarchical Validation, Repair, and Optimization
SKILLFOUNDRY validates skills hierarchically, checking both executable correctness and usefulness as agent abstractions. Failures trigger repair or optimization loops, while system and synthetic-data tests address different scientific deployment conditions.
- Execution Validation: Execution testing checks whether a skill’s declared contract is executable through its resolved smoke target or registered test command.Failures initiate diagnosis, revision, and rerunning of the check.
- Utility Validation: Usefulness testing compares skill-enabled task performance with an ad hoc no-skill baseline.Weak or inconsistent advantages trigger an optimization loop that revises the skill.
- Scientific Validation Surfaces: System testing supports infrastructure-dependent deployments, including SLURM, environment modules, and cluster-specific execution conventions.
- Scientific Validation Surfaces: Synthetic-data testing supports regression checks when live resources are unavailable, unstable, rate limited, or expensive.
A.1.4 Novelty Review and Tree Update
SKILLFOUNDRY reviews candidate skills for novelty and redundancy after validation, then updates the domain tree through acceptance, merging, review, deprioritization, or pruning. Its implementation separates parallel exploration from controlled integration and evaluates the resulting skills on scientific workflows, including scDRS.
- A.1.4 Novelty Review and Tree Update: Candidate skills undergo novelty and redundancy review against the local library and external skill ecosystems before acceptance.The review considers lexical similarity alongside topic placement, provenance, intended scope, and marginal value.
- A.1.4 Novelty Review and Tree Update: Validation outcomes update the domain tree by accepting new leaves, merging duplicates, marking candidates for review, or deprioritizing low-value branches.Repeated failures leave branches partially covered, while redundant or low-value skills can trigger consolidation or pruning.
- A.1.4 Novelty Review and Tree Update: Exploration may run in parallel, but shared registries, reports, and site summaries remain refresh-owned to preserve coherent repository state.Only leaf-owned artifacts are merged automatically from isolated worker workspaces.
- A.1.4 Novelty Review and Tree Update: The staged pipeline maps resource search, skill extraction, testing, repair, benchmarking, optimization, refresh, and novelty checking onto domain-tree expansion and refinement.Stage-conditioned prompts and hierarchical evaluation support iterative mining, targeted repair, utility benchmarking, and novelty assessment.
- A.1.4 Novelty Review and Tree Update: SKILLFOUNDRY evaluates skills on scDRS by comparing three Biomni runs with skills against three without skills and an expert reference analysis.The evaluation combines qualitative criteria with quantitative agreement measures, including RMSE.
- A.1.4 Novelty Review and Tree Update: With SKILLFOUNDRY skills, Biomni satisfies all seven qualitative scDRS criteria, matches expert scores in two of three runs, and lowers mean RMSE from 0.11 to 0.02.The with-skill workflow also produces richer figures that summarize FDR-supported signal and within-cell-type heterogeneity.