PaperlayerView exact revision on arXiv

Source-linked AI summary

Scientific Agent Skills: A Library of Procedural Knowledge for Research Agents

Timothy Kassis, Vinayak Agarwal, Yuhuan He, Darshil Patel, Aubrey M. Brueckner

arXiv:2609.00065v2cs.CLcs.AI

TL;DR

Scientific Agent Skills addresses the gap between runnable code and defensible scientific analysis by packaging field-specific procedures in an openly licensed library of 163 skills. The paper measures the library’s documentation footprint and lexical separability, finding low resident cost and manageable instruction-file workflows, but no task-level evaluation and substantial overflow when all references are loaded.

  • Problem

    Agents need field-specific conventions for tests, identifiers, and result caveats because this procedural knowledge is distributed across documentation, standards, guidelines, and tacit practice.

  • Method

    The paper builds and describes 163 portable skill directories with human-readable instructions, optional references and scripts, then recomputes documentation and lexical measurements from a pinned release.

  • Results

    7.1% of a 200,000-token window holds all resident descriptions, while the median documented workflow uses 23.9%; loading every reference file makes 29 of 46 workflows exceed the window.

  • Takeaways & Limitations

    Progressive disclosure can offer 163 specialised procedures without filling standing context, while workflows require selective reference loading.

  • Takeaways & Limitations

    The paper reports no task-level evaluation or host selection rate, and its measurements describe documentation layout rather than whether agents select or follow skills correctly.

Abstract

from arXiv · show

A language-model agent asked to analyse an experiment will usually return working code. Whether the analysis is defensible is a different question. A defensible analysis depends on procedural choices: which test the field accepts, which identifier namespace is authoritative, and which caveats must accompany a result. We present Scientific Agent Skills, an open library of 163 such procedures in 16 areas of practice, including genomics, cheminformatics, medical imaging, study design and scientific communication. Each skill is a directory built around a versioned, human-readable instruction file. An agent loads the file only when a task calls for it; the directory often also contains reference material and runnable scripts. We report no task-level evaluation and no host selection rate. We measure two properties of the documentation corpus: the always-resident descriptions of all 163 skills cost 7.1% of a 200,000-token window, and the median documented workflow fits within 23.9% of it, although 29 of 46 would overflow if every reference file were loaded. Openly licensed and available at https://github.com/K-Dense-AI/scientific-agent-skills.

1 Introduction

Scientific Agent Skills addresses the procedural gap between code that runs and analyses that are defensible. It records field-specific conventions in portable, reviewable skill directories, while this paper describes the library and corpus measurements rather than validating task-level effects.

  • Motivation: Agents can produce plausible code while violating multiple-testing, replication, technical-confounding, or coordinate conventions.The paper links these procedural errors to documented standards and prior evidence.
  • Motivation: Scientific analysis requires procedures specifying applicable tests, authoritative identifier namespaces, and result caveats.The paper notes that this knowledge is distributed across documentation, guidelines, standards, and tacit field practice.
  • Contribution: Each skill is a portable directory with a human-readable instruction file, optional references and scripts, and no required runtime, API, or service.Hosts retain names and short descriptions in context and load full instructions when relevant.
  • Contribution: Scientific Agent Skills is an openly licensed library of 163 procedures spanning biology, chemistry, medicine, physical sciences, and scientific communication.The library directly addresses examples from the motivation through rules for correction, experimental design, RNA-seq processing, and coordinate conventions.
  • Scope: The paper measures documentation properties rather than agent performance, selection rates, or independent validation of the collection.The authors recompute corpus quantities at a pinned release, while broader independent evaluation remains necessary.

2 What an agent skill is

A skill is a structured, human-readable directory whose content is progressively disclosed: descriptions remain resident, instructions load when relevant, and references load only when requested. This reduces standing-context cost but does not establish correct selection or execution.

  • Skill structure: A skill minimally contains SKILL.md with a constrained YAML header and Markdown instructions written for an agent.The specification also recommends references/, scripts/, and assets/ directories.
  • Skill maintenance: Authoring combines model drafting, manual review, available-environment workflow tests, revision, and feedback, but these steps are not domain validation.The paper distinguishes maintenance checks from evidence that a skill works scientifically.
  • Progressive disclosure: The disclosure model keeps every skill’s name and description resident, loads instruction files when relevant, and defers reference files until pointed to.The three tiers differ by orders of magnitude in size.
  • Corpus costs: 14,246 resident tokens cover all 163 skills, equal to 7.1% of a 200,000-token reference window.The complete instruction corpus totals 482,506 tokens, while 83.7% of documentation remains unread unless activated instructions point to it.
  • Scope: Tiering defers documentation cost but leaves routing dependent on resident descriptions and does not show that agents select correctly.The measurements describe repository layout rather than agent traces.

3 What the library covers

The 163-skill library spans 16 reporting categories and combines package, retrieval, platform, and method-oriented procedures. Its worked examples show cross-category composition, with communication and statistics linking domain skills.

  • Taxonomy: 163 skills span 16 categories under a reporting taxonomy assigned by the authors rather than enforced by the repository.Each skill receives exactly one primary category for reporting, although the library’s own headings overlap.
  • Skill types: The library includes package workflows, data-retrieval skills, research-platform and laboratory-automation skills, and method-and-judgment skills.Examples cover Scanpy, PyDESeq2, identifier resolution, study design, test selection, assumptions, and uncertainty.
  • Data retrieval: Database-lookup documents 78 databases across 9 domains, including 57 with no credential requirement and 18 requiring a key.The source passage also notes that 3 databases are paid or otherwise restricted, each with a named free alternative.
  • Composition: 46 worked examples have a median span of 4 categories, while 81 of 162 named skills appear in exactly one example.Scientific-visualization and scientific-writing are the most frequently named skills, appearing in 27 and 26 examples respectively.
  • Composition: Communication and statistics skills serve as connective tissue linking domain skills in the documented examples.This describes the authors’ worked examples, not observed agent sessions.

4 Can descriptions distinguish among skills?

The resident descriptions are usually lexically distinct, but a small upper tail contains adjacent skills that require explicit routing guidance. Cross-references target many close pairs, though coverage remains incomplete and the measure is only a lexical proxy.

  • Measurement: The authors represent descriptions with filtered TF-IDF tokens and calculate cosine similarity across all 13,203 pairs as a lexical confusability proxy.This preprocessing convention does not establish that a host would confuse any pair.
  • Lexical separation: The median nearest-neighbour cosine is 0.15 and the ninetieth percentile is 0.26, indicating that most descriptions lack a lexically close competitor.The closest pair, datamol and rdkit, reaches 0.40.
  • Routing guidance: 33 descriptions contain 56 pointers connecting 44 distinct pairs, including 12 reciprocal pairs.These pointers act as resident-tier routing hints for neighbouring skills.
  • Routing guidance: Pointer-linked pairs have mean cosine 0.185 versus 0.013 for unlinked pairs, and pointers cover 17 of the 20 closest pairs.The enrichment is concentrated among the most similar descriptions.
  • Limitations: The closest unguarded pair reaches cosine 0.36, showing that manually authored cross-references do not cover every high-overlap collision.Coverage depends on whether authors notice each collision, so repeated release-level measurement could identify remaining gaps.
  • Limitations: The security scanner flags 301 pairs but catches none of the 20 closest pairs, so its heuristic is not a substitute for lexical similarity analysis.Its flagged pairs have mean cosine 0.020 versus 0.013 for unflagged pairs.

5 Do the skills a workflow needs fit in context?

The documented workflows fit within the window when resident descriptions and instruction files are loaded, but loading every reference file exceeds the window for many workflows. Progressive disclosure therefore preserves context capacity while requiring selective reference loading for demanding workflows.

  • 472 skill slots across 46 examples cover 162 of 163 skills and define the documented workflow set used for context-cost analysis.The examples are treated as sessions translating per-skill token counts into context-window costs.
  • 47,706 tokens is the median workflow cost with resident descriptions and instruction files, or 23.9% of a 200,000-token window; none of 46 workflows exceeds it.The largest workflow costs 62,476 tokens under this loading scheme.
  • 225,498 tokens is the median cost when every reference file is loaded, or 112.7% of the window; 29 of 46 workflows exceed it.The largest workflow reaches 466,131 tokens, making the all-references series an upper bound by construction.
  • The median workflow can load 85% of its own reference material, while 17 of 46 workflows can load all references and 5 can load under half.The lower quartile can afford 61%, and the most demanding workflow 35%.
  • The reported totals understate session context because they exclude scripts, assets, tool output and host wrappers, while retaining a duplicated median 67-token description.The duplicate is intentionally retained because it biases the total upward.
  • Installing only project-relevant skills bounds standing context and routing candidates, but whether this improves selection accuracy remains untested.Every installed skill adds an always-resident description even when unused.

6 Limitations

The paper’s limitations concern validation, selection measurement, execution and context sufficiency, portability, review coverage, measurement conventions, and the fixed release snapshot. These boundaries mean the corpus measurements do not establish scientific benefit, reliable host behavior, or comprehensive currency.

  • No task-level evaluation shows that giving agents these skills improves scientific work.A controlled study varying only installed skills and scoring outcomes against published conventions is still needed.
  • Lexical similarity measures are proxies for selection and do not estimate host selection error rates.Estimating selection accuracy would require labelled task-to-skill pairs, a fixed host and a scored selection step.
  • A selected skill can still fail scientifically because execution, study conditions, contextual sufficiency and dependency currency are not measured or fully audited.Skills structure scientific judgment but do not replace expert review.
  • Host portability is intended rather than tested, because wrappers, metadata, tools and permission models can differ across hosts.A compatibility claim would require a host-by-skill matrix that the paper does not provide.
  • Review is uneven: 32 of 163 skills do not list the authors, and automated structural checks cover only 105 skills.Two broken local paths remain outside the CI gate.
  • The measurements depend on stated lexical, parsing, threshold and tokenizer conventions that cannot be reproduced exactly from the published word lists.The paper describes each rule, while Appendix G varies routing rules and reports their effects.
  • All counts describe the pinned v2.65.0 snapshot, so later releases can differ.The authors state the pin and regenerate each number from the repository tree.

7 Availability and citation

The library is openly available under the MIT licence, and the paper identifies the exact release and commit used for its measurements. Reuse should cite both the release tag and the specific skill metadata, while the reported reach estimate is not evidence of use or benefit.

  • Scientific Agent Skills is openly available under the MIT licence at the project repository.The described release is tag v2.65.0 at commit f6fcafeb1cc8c82eca0160a18bc41c38427b8e0f.
  • The README names Claude Code, Claude Cowork, Codex, Gemini CLI, Google Antigravity and Cursor as supported hosts, subject to qualification in Section 6.The paper recommends installation with npx skills add K-Dense-AI/scientific-agent-skills.
  • A reproducible citation should include the release tag, the skill name and its independently maintained metadata.version.Clinical, regulatory and imaging uses should also state the relevant skill’s scope limits.
  • The paper releases no analysis package, but corpus-derived quantities can be recomputed from the public repository tree at tag v2.65.0.Recomputation does not validate the authors’ labels or interpretation.
  • The token counts use o200k_base and documentation text only, so hosts using another tokenizer obtain different absolute figures while tier ratios remain stable.Scripts, assets, host wrappers and system prompts are excluded from the token corpus.

A.1 The full token ledger

The library’s tiered disclosure model keeps the complete corpus outside a single 200,000-token window while making individual activated skills manageable. Its contents vary widely, and instruction files emphasize recurring procedural roles and concrete workflows.

  • Token ledger: 14,246 tokens, or 7.1% of a 200,000-token window, hold the always-resident descriptions of all 163 skills.Inlining complete instruction files would require 482,506 tokens, while the full documentation corpus would need 14.8 such windows.
  • Token ledger: 17,103 tokens is the median session total when one activated skill contributes its complete instruction file; loading all references raises that median to 29,440 tokens.The heaviest skill reaches 231,263 tokens and exceeds the reference window by itself.
  • Library contents: 154 of 163 skills ship references, 105 contain 651 files under scripts/, and 34 ship static assets.The 651 script-directory files include 527 Python files, 2 shell scripts, and 122 XML or schema support files.
  • Instruction files: The instruction corpus contains 844 distinct level-2 headings, with a median of 10 per file; 104 skills state when they apply, 126 give procedures or decision rules, and 84 include validation checks or caveats.The role grouping uses a keyword rule over headings, so unmatched headings are not counted.
  • Instruction files: 285 lines is the median complete file length against the 500-line recommendation, and the library contains 1,489 fenced code blocks.Of those blocks, 760 are Python, 625 are shell, and 67 declare no language; these counts measure worked commands rather than prose volume.

B.2 Declared capability and provenance

The library records declared tool expectations and provenance for its skills, while portability depends on host-specific interpretation. Its repository layout separates agent payloads from supporting infrastructure, with documented scope and validation boundaries.

  • Declared capabilities: 100 of 163 skills declare allowed tools, most commonly file reading (99), shell access (94), file writing (92), and editing (78).The field is optional and advisory, so undeclared tools may still be required.
  • Provenance: All 163 skills carry individual version strings, while authorship is recorded for 159 and attribution covers 131 internal, 24 external, 4 vendored, and 4 unattributed skills.The paper does not treat version distributions as maturity measures, and four skills leave license metadata unset.
  • Portability: Portability means conformance to a shared file layout rather than identical behavior, because hosts differ in installation, discovery, and supported metadata fields.The paper does not test a host-compatibility matrix.
  • Portability: 25 of 163 skills carry host-specific metadata mappings, and 116 declare compatibility strings naming environment requirements.The validator accepts these mappings even though the specification defines metadata values as strings, so hosts may consume, ignore, or reject them.
  • Repository structure: The repository separates agent payloads under skills/<name>/ from tests, shared checks, generated diagrams, and CI workflows elsewhere.Only each skill’s name and description are resident before selection; the shipped payload includes SKILL.md and permitted supporting directories.

E How the library is validated

Validation combines all-skill structural checks with narrower script-skill tests, selected behavioral suites, and security scanning. These checks describe repository conformance and scanner output, not scientific correctness or a vulnerability census.

  • Structural conformance: All 163 skills undergo frontmatter checks, while the 10-check structural contract covers only the 105 script-bearing skills.The narrower contract leaves 58 skills outside those pull-request checks.
  • Structural conformance: The full-tree structural rerun found 2 broken local paths in documentation-only skills outside the CI loop.This demonstrates that pull-request scope changes what CI catches.
  • Test coverage: 105 script-bearing skills have 105 suites across 129 test files, but only 20 suites run on pull requests and 85 require a local isolated runner.Suite existence does not establish behavioral coverage, and hosted runners lack some scientific dependencies.
  • Interpretation: None of the repository gates establishes that following a skill produces correct science or validates the substance of its guidance.The gates establish frontmatter, structural, or asserted behavioral properties within their respective scopes.
  • Security scanning: The published security scan reports 988 findings, including 34 critical findings, but the counts are scanner output rather than a vulnerability census.A prior triage covered 33 critical findings from an earlier 154-skill, 817-finding report, leaving the remaining findings at this pin untriaged.
  • Security scanning: Cross-skill rules account for 449 findings and can flag structural coincidences, so the paper does not convert scanner counts into vulnerability counts.The dominant rules include complementary trigger conditions and broad potential relay chains.

F What the skills claim, and what they decline

The library limits high-stakes uses through scope clauses, but those limits are unevenly stated in the instruction files agents actually load. Confirmed clauses are concentrated in clinical, regulatory, translational, and medical-imaging skills.

  • Stated boundaries: The library frames regulatory and standards outputs as prepared for qualified review, while imaging and pathology work is research-only.The standards-readiness skill explicitly excludes compliance, certification, and accreditation decisions.
  • Scope limits: 10 of 163 instruction files contain manually confirmed scope-limiting clauses, with 5 of 9 high-stakes clinical, regulatory, and translational skills and 2 of 6 imaging-related skills represented.The confirmed set is a lower bound because lexical search may miss differently worded clauses.
  • Stated boundaries: Category-level README caveats do not uniformly appear in individual instruction files, so users installing a single skill may not encounter them.The agent loads the activated instruction file rather than the README’s category framing.

G Routing sensitivity

Routing-sensitivity analysis tests whether description similarity depends on preprocessing choices and compares the closest-description pairs with a security scanner rule. The closest pairs remain stable across alternatives and largely differ from scanner-flagged pairs.

  • Preprocessing sensitivity: At least 9 of the original 10 closest description pairs remain in the top-10 set under each preprocessing alternative.The alternatives vary stop-list and tokenization choices, supporting stability of the named collisions.
  • Scanner comparison: The security scanner flags 301 pairs with mean cosine 0.020, versus 0.013 for unflagged pairs, and overlaps 0 of the 20 closest pairs.The scanner’s complementary-description rule therefore identifies a largely different set from the routing-similarity ranking.

H Estimating the installed base

The paper estimates the installed base from GitHub cloning activity using a model-based sampling-rate calculation, while bracketing the result with directly observed cloners and page visitors. The estimate is constrained by a non-reproducible traffic window and assumptions about heterogeneous cloning behavior.

  • The traffic result is a dated snapshot that reviewers cannot reproduce because GitHub exposes only a rolling 14-day window to accounts with push access.The authors report it instead of a company-reported user count with no stated method and state the estimate’s assumptions for scrutiny.
  • 161,210 clones from 15,682 distinct cloners were recorded during the 2026-08-14 to 2026-08-27 window, but clone volume is not a people count.Active cloners averaged roughly 8 clones per active day, suggesting continuous integration and mirroring alongside installation activity.
  • D = 20,234 daily unique cloner counts and P = 15,682 window-level unique cloners yield k = D/P = 1.29 mean active days among observed cloners.The estimator models each actor as cloning independently on each day with probability p.
  • Heterogeneous cloning behavior can inflate k, raising the implied observation rate q and lowering the estimated pool N.The paper presents this downward bias as conservative for its interest in a larger installed-base estimate.
  • 15,682 distinct cloners provide a model-free floor, while applying the estimator to page visitors gives a higher estimate of 51,089.The 37,905-star count is within 6% of the clone-pool estimate and serves as a check rather than an input.
Loading 2609.00065v2…