Source-linked AI summary
SkillAlchemy: Open-World Agent Skill Creation
Hengjun Wang, Shuyue Wei, Boyi Liu, Jun Yang, Yongxin Tong
TL;DR
Reliable skill creation is difficult when expert knowledge, model priors, and execution traces are unavailable for unfamiliar tasks, even though useful procedures may exist in open-world materials. SkillAlchemy discovers omitted requirements, admits procedures only within evidence-supported scope, and compiles them into skills. Across 87 SkillsBench tasks, it improves pass rate over no-skill execution and the strongest automated baseline while performing comparably to human-curated skills.
Problem
Reliable skill creation lacks accessible procedural knowledge for unfamiliar tasks because existing approaches depend on experts, model priors, or archived execution traces.
Method
SkillAlchemy uses contrastive evidence to discover implicit requirements, admits procedures according to evidence-supported scope, and compiles admitted content into a skill package.
Results
19.9 percentage points over no-skill execution and 8.6 percentage points over the strongest automated baseline are SkillAlchemy's pass-rate improvements across 87 SkillsBench tasks, with performance comparable to human-curated skills.
Takeaways & Limitations
Reliable open-world skill creation should treat source knowledge as evidence admitted under explicit scope rather than instructions copied directly.
Abstract
from arXiv · showhide
Agent skills are reusable procedural artifacts that extend language agents with specialized workflows, tool conventions, and domain behaviors at inference time. However, creating reliable skills still depends largely on human authorship, model priors, or execution traces. These sources are often unavailable for unfamiliar tasks, suggesting the need to create skills from open-world materials. In this paper, we study open-world skill creation: given an underspecified skill brief and a source-access specification, a creator must discover behavior-relevant requirements omitted by the brief and determine how broadly each source-derived procedure is justified. We propose SkillAlchemy, an admission-centered framework for source-grounded skill creation. SkillAlchemy identifies implicit requirements through contrastive evidence, admits candidate procedures based on evidence-supported scope, and compiles the admitted content into a grammar-guided skill package. Extensive experiments across 87 SkillsBench v1.1 tasks demonstrate that SkillAlchemy improves pass rate over no-skill execution by 19.9 percentage points and the strongest automated baseline by 8.6 percentage points, while achieving performance comparable to human-curated skills.
1 Introduction
Open-world materials can supply procedural knowledge for unfamiliar tasks, but reliable skill creation must recover omitted requirements and justify the scope of source-derived procedures. SkillAlchemy addresses these challenges through evidence-centered admission and improves performance over automated alternatives.
- Motivation: Agent skills package reusable workflows and domain-specific behaviors that language agents can load and execute at inference time.They may include metadata, instructions, scripts, references, assets, and other on-demand resources.
- Motivation: Existing skill creation relies on experts, model priors, or archived execution traces, which are often unavailable for unfamiliar tasks.Open-world materials such as documentation, repositories, and issue reports may contain underused procedural knowledge for these tasks.
- Challenges: Open-world skill creation must discover operational requirements omitted by task briefs before acquiring evidence.Briefs leave implicit requirements, failure modes, and operational boundaries, while sources are organized around their own subjects rather than the complete target capability.
- Challenges: Open-world findings require scope justification because source-specific practices may be too narrow to become reusable instructions.Skill creation should promote consistent practices, retain context-bound ones as scoped examples, and exclude weakly supported or conflicting candidates.
- SkillAlchemy: SkillAlchemy frames creation as source-grounded procedure admission, combining implicit requirement discovery, grounded admission, and skill package compilation.Candidates become reusable instructions only when evidence supports both the procedure and its claimed scope.
- Results: 19.9 percentage points is SkillAlchemy's improvement in pass rate over no-skill execution across 87 SkillsBench tasks, while it exceeds the strongest automatic baseline by 8.6 points.The framework achieves performance comparable to human-curated skills.
2 Related Work
Prior work treats skills as reusable procedural abstractions and creates them through human authorship or interaction traces. SkillAlchemy extends broad-source creation by acquiring and aggregating evidence before turning source material into skills.
- Skills for Agents: Agent skills are reusable procedural artifacts rather than ordinary prompts or atomic tool calls.Prior work reports that reusable skill abstractions can improve agent performance on interactive tasks.
- Skill Creation: Skill creation methods include human-authored skills and interaction-trace-based libraries or lessons learned from task experience.Human-written and benchmark-provided skills encode procedural knowledge, while trace-based methods derive reusable content from interaction or experience.
- Broad-Source Creation: SkillAlchemy belongs to the broad-source creation line but performs knowledge acquisition and evidence aggregation before skill creation.This focus complements human-authored and trace-based creation rather than treating retrieved source content directly as skill content.
3 Method
SkillAlchemy treats open-world skill creation as a source-grounded procedure-admission problem: it discovers implicit requirements, evaluates candidate procedures at evidence-supported scopes, and compiles admitted content into an installable skill package.
- Framework overview: SkillAlchemy separates requirement discovery, procedure admission, and package compilation into sequential decisions for open-world skill creation.The framework first acquires evidence for omitted requirements, then determines supported scope, and finally renders the admitted content.
- Implicit requirement discovery: Stage 1 converts an underspecified brief into operational factors, focused research questions, and structured findings without treating brief-specific values as reusable requirements.A factor becomes implicit only when evidence shows that varying it changes procedural behavior.
- Evidence-grounded procedure admission: Stage 2 groups findings by procedural decision and induces candidates whose actions, recovery, verification, and applicability conditions are directly supported by evidence.The method preserves conditional cases and leaves unsupported components unspecified rather than generalizing beyond observed contexts.
- Evidence-grounded procedure admission: Admission assigns each candidate the widest evidence-justified scope and classifies it as general, scoped, or excluded based on support, consistency, and scope conditions.General candidates become reusable instructions, scoped candidates remain context-bound examples, and excluded candidates stay in the audit record.
- Skill package compilation: Stage 3 renders admitted instructions and scoped examples into a standard skill package without changing their admitted scope.The package centers on SKILL.md and may include references, scripts, assets, safeguards, and grammar-guided organizational patterns.
4 Experiments
SkillAlchemy is evaluated across 87 SkillsBench v1.1 tasks and four agent–model configurations against no-skill, human-curated, and automated skill baselines. It improves aggregate performance, shows domain-level variation, incurs intermediate creation cost, and benefits from all tested components while resisting source perturbations.
- Overall Performance: SkillAlchemy exceeds no-skill execution by 19.9 percentage points and the strongest automated baseline, MUSE-Autoskill, by 8.6 percentage points.It achieves the highest overall performance in three of four agent–model configurations.
- Overall Performance: 55.8% observed avg@5 places SkillAlchemy 1.5 percentage points above the Human-Curated Skill.The comparison is reported at the aggregate level across the evaluated settings.
- Domain-Level Analysis: SkillAlchemy performs better than the human-curated skill in Finance and Economics, Software Engineering, and Office Tasks, while Media shows a substantial deficit.The Media gap is associated with less consistent preservation of failure-prone steps and calibrated parameter choices.
- Statistical Uncertainty: 1,740 binary evaluations per skill setting yield 55.8% avg@5 for SkillAlchemy, versus 54.4% for Human-Curated Skill, 47.2% for MUSE-Autoskill, and 46.0% for OpenSkill.The reported 95% Wilson intervals quantify pooled within-setting uncertainty, so the 1.4-point human-curated margin is interpreted descriptively.
- Creation and Execution Cost: 23.21 minutes per task makes SkillAlchemy slower to create than the two Skill-Creator baselines but faster than OpenSkill and MUSE-Autoskill.Its downstream execution uses 709K tokens and 6.39 minutes per run, with latency comparable to other methods.
- Component Ablation: Every one-component ablation reduces avg@5 across Software Engineering, Office, and Natural Science.The ablations remove implicit requirement discovery, structured findings, procedure admission, or grammar-guided rendering under the same source scope and creation budget.
- Robustness under Source Perturbations: SkillAlchemy promotes none of 12 injected irrelevant, conflicting, or adversarial payloads and retains 17–18/20 downstream passes across all perturbation conditions.Across four baseline creators, conflict promotes 9/16 payloads and reduces pooled pass count from 58/80 to 35/80.
5 Conclusion
SkillAlchemy frames reliable open-world skill creation as evidence-based admission under explicit scope. Across 87 SkillsBench v1.1 tasks, it outperformed both no-skill execution and the strongest automated baseline while matching human-curated skills in aggregate.
- 5 Conclusion: SkillAlchemy addresses omitted behavior-changing requirements and restricts source-derived procedures to evidence-supported scope.Its pipeline combines implicit requirement discovery, evidence-grounded procedure admission, and scope-preserving skill-package compilation.
- 5 Conclusion: 19.9 percentage points: SkillAlchemy improved pass rate over no-skill execution across 87 SkillsBench v1.1 tasks and four agent–model configurations.The reported improvement is measured across the paper’s full task and configuration set.
- 5 Conclusion: 8.6 percentage points: SkillAlchemy improved pass rate over the strongest automated baseline, reaching aggregate performance comparable to human-curated skills.The conclusion presents both the automated-baseline comparison and the human-curated comparison as aggregate outcomes.
- 5 Conclusion: Reliable skill creation should treat open-world knowledge as evidence admitted under explicit scope rather than instructions copied directly.This conclusion follows the paper’s admission-centered framing of source-derived procedures.
A Experimental Protocol
This section documents the experimental controls supporting the reported results, covering baseline reproduction, evaluation isolation, and shared runtime configuration.
- A Experimental Protocol: Baseline reproduction is documented as part of the experimental protocol.The section states that it documents baseline reproduction behind the reported results.
- A Experimental Protocol: Evaluation isolation is documented to separate evaluation conditions from the experimental protocol.The section explicitly identifies evaluation isolation as one documented component.
- A Experimental Protocol: Shared runtime configuration is documented as supporting the reported experimental results.The section identifies shared runtime configuration alongside baseline reproduction and evaluation isolation.
A.1 Baseline Implementation Details
The baseline implementations reproduce OpenSkill’s search, creation, verification, and refinement procedure, and MUSE-Autoskill’s task-level skill distillation and installation procedure.
- A.1 Baseline Implementation Details: OpenSkill reads visible task information, performs creation and independent verification searches, then plans and creates a skill.It evaluates the skill with a virtual verifier before any refinement step.
- A.1 Baseline Implementation Details: OpenSkill diagnoses failed virtual tests as skill defects or knowledge gaps and refines the skill for up to three rounds.This refinement follows the reported baseline procedure.
- A.1 Baseline Implementation Details: MUSE-Autoskill distills reusable procedures, key operations, validation steps, and common errors into a task-level skill installed without modification.The resulting skill is used for downstream evaluation after installation.
A.2 Evaluation Isolation Protocol
The evaluation protocol separates skill creation from downstream evaluation and excludes evaluation-only information from automated creation. Web access follows a common retrieval policy, while runtime versions and commit identifiers are pinned.
- A.2 Evaluation Isolation Protocol: Skill creation produces an installable SKILL.md package from permitted task briefs and source materials, whereas a fresh downstream agent performs evaluation.A benchmark verifier scores the downstream agent’s resulting submission.
- A.2 Evaluation Isolation Protocol: Evaluation-only assets, including held-out inputs, oracle artifacts, and verifier logic, are withheld from evaluated agents and excluded from automated skill creation.The protocol distinguishes four evaluation-only asset types and applies the exclusions to all automated methods.
- A.2 Evaluation Isolation Protocol: The Human-Curated Skill is mounted only during its evaluation condition and is never supplied as a creation source.This preserves the separation between creation inputs and the human-curated evaluation condition.
- A.2 Evaluation Isolation Protocol: Web-enabled creation excludes SkillsBench-related pages consistently across automated methods to maintain a common source policy and evaluation setup.The exclusion is applied specifically to retrieval during Web-enabled creation.
- A.2 Evaluation Isolation Protocol: Model endpoints, agent versions, adapter versions, and the SkillsBench commit are pinned across experiments.The protocol records exact identifiers for the endpoints, agents, adapters, and runtime commit.
B Generated Skill Artifacts Analysis
Skill packages keep runtime-facing guidance focused while organizing supporting evidence and resources separately. SkillAlchemy uses a corpus-derived grammar to express admitted procedures without adding or broadening task knowledge.
- Main-File Scope: Median top-level SKILL.md length is approximately 144–157 lines across agent–model configurations.Figure A.1 reports main-file length distributions for four configurations.
- Package Organization: Skill-creator baselines package reusable operations as generated scripts, whereas SkillAlchemy places procedures in SKILL.md and supporting details in scoped references.The comparison concerns package composition and runtime-facing organization.
- Grammar-Guided Compilation: The grammar is corpus-derived and guides package organization; it does not supply task knowledge or override evidence-based admission.It separates presentation decisions from admission of task-specific knowledge.
- Operational Grammar: The grammar covers activation, procedure structure, conditions, safeguards, verifiable outputs, resource placement, and progressive disclosure.Its component choices include triggers, ordered steps or decision structures, applicability boundaries, checks, and package-relative references.
- Grammar-Guided Compilation: Grammar-guided rendering selects executable and inspectable presentations while preserving applicability conditions, safeguards, and optional resources behind package-relative references.The grammar changes organization and presentation rather than task-specific procedures, scope, or supporting evidence.
D.1 Execution Case Study
The execution case study shows that near-complete criterion coverage and structural validity can still miss decisive numerical or objective-quality requirements. Matched cases therefore motivate explicit validation of task-determining computations.
- Matched Comparison: 33 matched SkillsBench tasks span 3 easy, 21 medium, and 9 hard tasks across seven conditions.The matched set contains valid task-level and verifier-criterion results for all conditions.
- Case 1: Numerical conditioning: In numerical conditioning, Human-Curated Skill, OpenSkill, and MUSE-Autoskill pass 5/5 runs, SkillAlchemy passes 4/5, and No Skill passes 3/5.All seven conditions satisfy at least eight of nine verifier criteria in every matched run.
- Case 1: Numerical conditioning: Edge-only power spectral density estimation changes the recovered SNR scale and selected masses, isolating numerical conditioning rather than artifact-format failure.The case requires reporting SNR and total mass for each waveform approximant.
- Case 2: Objective quality: In objective quality, No Skill, Human-Curated Skill, OpenAI Skill-Creator, and SkillAlchemy pass 5/5 runs, while Anthropic Skill-Creator passes 3/5.All outputs are feasible and internally consistent, but some runs exceed the objective-quality threshold.
- Cross-Case Takeaway: Structural validity and near-complete criterion coverage do not guarantee task success.Numerical pipelines need checks on decisive computations, while optimization tasks need objective-quality checks beyond feasibility.
E.1 Human-Curated Skill
The human-curated skill provides domain heuristics and a calibration workflow, but several local defaults, thresholds, and implementation details lack sufficient scope or executable grounding.
- Overview: The human-curated skill offers useful domain heuristics but includes several unscoped local rules.The artifact contains 92 source lines.
- Scope Boundaries: The benchmark’s 2°C acceptance threshold is presented as a general definition of good GLM calibration.The source labels this as an over-specific threshold and repeats it as a universal stopping or quality rule.
- Unscoped Defaults: Fixed parameter values and ranges are presented without specifying the lake, GLM configuration, or evidence boundary.The listed parameters include Kw, coef_mix_hyp, wind_factor, lw_factor, and ch.
- Reusable Abstraction: Parameter-effect relations transfer more safely than fixed values and can support residual-driven calibration.The skill contrasts reusable relations with local numeric defaults.
- Implementation: The displayed calibration code is not directly executable because modify_nml, calculate_rmse, sim_df, and obs_df are undefined.The surrounding workflow describes optimization and manual calibration, but the shown code omits these definitions.
E.3 OpenAI Skill-Creator
The OpenAI Skill-Creator artifact provides a runnable Lake Mendota calibration loop with conservative parameter iteration and scoring. Its fixed matching and coordinate-selection rules nevertheless require validation against task-specific semantics.
- Workflow: The skill turns the Lake Mendota sandbox into a reproducible calibration loop requiring a successful GLM run, output.nc, and temperature scoring.It targets calibration and validation using glm3.nml, forcing CSVs, observations, and output/output.nc.
- Calibration Procedure: The workflow confirms inputs, inspects the namelist, runs a baseline, scores it, and tunes small groups of physically meaningful parameters.Candidate families include light attenuation, vertical mixing, surface forcing, and drag or transfer coefficients.
- Safety Pattern: Candidate lifecycle combines backup, targeted patching, stale-output removal, execution, and scoring.The runner applies settings, clears stale output.nc, runs GLM, and prints RMSE when output is produced.
- Scoring: The scorer matches observations within 12 hours, interpolates model profiles to observation depth, skips missing pairs, and reports RMSE.It can override inferred variable and coordinate names when the NetCDF schema differs.
- Limitations: The fixed 12-hour matching rule may change both the matched observations and the reported RMSE because it is not task-stated or cadence-derived.Coordinate interpretation also remains a separate validation concern.
E.4 OpenSkill
OpenSkill provides a reusable GLM calibration workflow spanning setup inspection, bounded parameter search, validation, diagnostics, and reproducible delivery. Its broad structure remains useful, but task-specific bindings and exact matching policies may require reconstruction.
- Calibration procedure: The workflow establishes a baseline, validates inputs and outputs, edits parameters, runs GLM, computes RMSE, diagnoses residuals, and tracks candidates.It includes parser-based namelist editing, time-axis checks, matched-pair computation, diagnostics, and search termination conditions.
- Validation: Candidates are rejected when validation, execution, result-variable, time-axis, matched-pair, or coordinate checks fail.These checks prevent invalid simulations from being treated as calibration evidence.
- Reusable workflow: OpenSkill captures a robust calibration lifecycle without hard-coding a final answer.The workflow covers repeatable calibration while preserving reusable structure.
- Search constraints: The skill requires physically meaningful bounds, normalized parameter scales, and diagnostics beyond the primary metric.Secondary diagnostics include bias, seasonal behavior, depth-specific errors, group-wise RMSE, and matched-pair counts.
- Scope: OpenSkill is broadly operational but remains under-instantiated for concrete tasks and leaves exact-time matching policy underspecified.A solver may still need to reconstruct paths, dates, output names, schemas, and task-specific evidence.
E.6 SkillAlchemy
SkillAlchemy turns open-world skill creation into an evidence-grounded process that discovers implicit requirements, scopes procedures, and compiles reusable content while preserving task-local bindings. Its synthesis adds operational safeguards that address plausible false success and delivery validity.
- Operational loop: The framework covers a closed calibration loop from editing glm3.nml through execution, temperature extraction, scoring, residual diagnosis, and selecting the next parameter.The concrete commands instantiate this loop for the visible Lake Mendota task.
- Scope control: The skill preserves task-scoped bindings rather than treating paths, dates, targets, and candidate parameters as reusable defaults.New GLM setups require those bindings and model conventions to be revalidated.
- Implicit requirements: SkillAlchemy identifies implicit execution requirements beyond the task brief, including coordinate semantics, unreliable exit codes, and working-directory-dependent paths.These omissions can produce plausible-looking false success.
- Delivery validity: Delivery requires a runnable final artifact, adequate coverage, physical validity, and a clean rerun rather than merely a low score.The discovered gate addresses the difference between a requested final namelist and evidence establishing a valid delivered result.
- Evidence synthesis: SkillAlchemy converts evidence into reusable operation models for execution, scoring, calibration, and safe editing.The evidence is organized into models rather than copied as universal constants.