PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillEvolver: Skill Learning as a Meta-Skill

Genrui Zhang, Erle Zhu, Jinfeng Zhou, Caiyan Jia, Hongning Wang

arXiv:2605.10500v1cs.AI

TL;DR

Static and one-shot skills lack a mechanism to improve from real use, while existing acquisition methods often assume substantial offline preparation. SkillEvolver uses a portable meta-skill to author, deploy, and refine domain-skill artifacts from fresh-agent behavior without updating model weights, achieving higher benchmark performance than curated and no-skill baselines.

  • Problem

    Existing skills are authored once, while one-shot generation lacks grounded feedback and trace- or RL-based methods assume substantial offline preparation for each domain.

  • Method

    SkillEvolver uses a meta-skill to author, deploy, and refine reusable domain skills from separate agents’ behavior, with fresh-agent auditing and no model-weight updates.

  • Results

    56.8% accuracy on 83 SkillsBench tasks exceeds 43.6% for curated human skills and 29.9% for the no-skill baseline.

  • Takeaways & Limitations

    SkillEvolver reframes skill creation as artifact-level adaptation that can produce reusable procedural skills through bounded deployment-time trials.

  • Takeaways & Limitations

    The headline results use a single configuration, Claude Opus 4.6 + Claude Code, so cross-LLM parity is a design property rather than a benchmark-scale empirical finding.

Abstract

from arXiv · show

Agent skills today are static artifact: authored once -- by human curation or one-shot generation from parametric knowledge -- and then consumed unchanged, with no mechanism to improve from real use. We propose \textbf{SkillEvolver}, a lightweight, plug-and-play solution for online skill learning, in which a single meta-skill iteratively authors, deploys, and refines domain-specific skills. The learning target of SkillEvolver is the skill's prose and code, not model weights, so that the resulting artifact drops into any agent without retraining; and the meta-skill itself is just another skill, loaded through the same interface by any protocol-compliant CLI-agent. Unlike trace-distillation, the meta-skill refines only after deploying the learnt skill, such that the learning signal comes from failures another agent encounters while using it -- not from exploratory traces alone. Refinement iterations are governed by a fresh-agent overfit audit that catches possible leakage as well as deployed-skill-specific failures, including the silent-bypass mode in which a skill appears valid in content but is never invoked at runtime. On $83$ SkillsBench tasks spanning $15^{+}$ domains, SkillEvolver reaches $56.8\%$ accuracy versus $43.6\%$ for curated human skills and $29.9\%$ for the no-skill baseline; on three GPU kernel optimization tasks from KernelBench, it also raises mean speedup from $1.16$ to $1.51$ on average.

1 INTRODUCTION

SkillEvolver targets online learning of reusable domain skills under few-trial, on-demand conditions, addressing limitations of one-shot generation and offline trace- or RL-based acquisition. It deploys candidate skills to fresh agents and refines them from observed use.

  • Agent skills encode procedural knowledge as reusable artifacts containing instructions, scripts, references, examples, and constraints.
  • One-shot parametric skill generation lacks grounded feedback and can perform no better than, or worse than, using no skill.
  • Trace- and RL-based methods rely on substantial offline preparation, whereas many real-world tasks arrive individually with only a handful of affordable exploration trials.
  • SkillEvolver adapts an external skill artifact for a new task through exploration, authoring, deployment to fresh agents, and refinement from their behavior rather than model-parameter updates.
  • 56.8% accuracy on 83 SkillsBench tasks exceeds 43.6% for curated human skills and 29.9% for the no-skill baseline.

2 RELATED WORK

Related work includes automated skill creation, inference-time agent memory, trace distillation, reinforcement learning, and broader agent scaffolds. SkillEvolver instead produces a transferable skill artifact for other agents to load.

  • ACE iteratively edits an in-context playbook, while SkillEvolver produces a self-contained domain skill that a different agent can later load.
  • MemGPT and generative agents equip an agent at inference time but do not author a transferable skill as a separate deliverable.
  • Trace2Skill mines around 200 trajectories per domain, while SkillRL pools experiences across many training tasks to grow a skill library.
  • Agent evaluation requires controlling whether training-label information leaks into artifacts that are later evaluated on test data.
  • SkillEvolver is orthogonal to tool-use models, code-as-action loops, multi-agent orchestration, and agent benchmarks because it produces artifacts those systems can leverage.

3 METHOD

SkillEvolver is a meta-skill that keeps the agent’s weights fixed while iteratively authoring, deploying, contrasting, patching, and auditing portable domain-skill artifacts. Its loop uses strategy-diversified trials and fresh-agent deployment to identify and correct guidance failures.

  • 3.1 SKILLEVOLVER AS A META-SKILL: SkillEvolver instructs a fixed CLI-agent to author, refine, and deploy reusable domain skills rather than modify itself or model weights.
  • 3.1 SKILLEVOLVER AS A META-SKILL: The learning signal comes from a separate Domain-Skill Agent’s behavior with the candidate skill, including what it does or fails to do.
  • 3.2.3 INDEPENDENT AUDIT AND FINALIZATION: Each iteration explores K training-time trials, analyzes traces, synthesizes a targeted revision, and audits it in an independent fresh session before acceptance.
  • 3.2.1 STRATEGY-DIVERSIFIED EXPLORATION: Strategy-diversified exploration assigns distinct high-level solution plans to fresh agents, covering axes such as library choice, algorithm family, and instruction interpretation.
  • 3.2.2 CONTRASTIVE SKILL UPDATE: For binary-reward tasks, the agent contrasts passing and failing trials; for scalar-reward tasks, it contrasts top- and bottom-scoring trials to identify missing guidance.
  • 3.2.2 CONTRASTIVE SKILL UPDATE: The resulting patch edits prose and code in the skill artifact, preserving working guidance while adding missing constraints, code patterns, or tools without touching weights.
  • 3.2.3 INDEPENDENT AUDIT AND FINALIZATION: The Auditor checks self-containment, trace grounding, abstraction from training constants, strategy abstraction, script visibility, and resistance to silent bypass.

4 EXPERIMENTS

The experiments compare SkillEvolver with no-skill, curated, self-generated, and ablated pipelines on SkillsBench and KernelBench, then analyze refinement, efficiency, category-specific gains, and failure modes. SkillEvolver performs best overall, with gains driven especially by refinement and by tasks where curated skills are ineffective.

  • Skill quality comparison: 56.87% avg@5 on SkillsBench exceeds the no-skill baseline by +27.0 percentage points and the human-curated skill by +13.3 percentage points.SkillEvolver (R=1) reaches 48.2%, while Self-Gen reaches 32.0% and SkillCreator-SkillsBench reaches 33.9%.
  • Skill quality comparison: 1.027 to 1.089, 1.117 to 1.218, and 1.326 to 2.226 mean speedup increases occur on deepnarrowmlp, shufflenet, and gru at R=2.KernelBench evaluates three GPU kernel optimization tasks using correctness-weighted speedup over five validation runs.
  • Skill quality comparison: 74.7% of SkillsBench tasks are wins or ties against the human-curated skill, with 24/83 wins, 38/83 ties, and 21/83 losses.The headline gain primarily widens the solvable set, while curated skills remain stronger on some highly domain-specific DSL or convention tasks.
  • Component ablation: 48.2% to 56.87% avg@5 follows from the second iteration, an improvement of +8.7 percentage points over the one-pass pipeline.At R=2, the revised skill is redeployed as a live dependency, followed by renewed exploration and patching.
  • Cost-quality trade-off: $3.92 per task yields the R=2 pipeline’s accuracy gains with only $0.28 (+8%) above R=1, while downstream validation uses −19.4% tokens, −15.3% turns, and −23.8% wall-clock versus no skill.SkillCreator-SkillsBench costs $6.97 per task and regresses on all three downstream efficiency metrics.
  • Per-category analysis: B2 (+60 pp), D (+40 pp), B3 (+33 pp), and C1 (+13 pp) show the largest category gains, concentrating improvements where curated skills fail hardest.On A, the agent already solves tasks without a skill, so the pipeline is not invoked; refinement adds most on D (+20 pp) and C1 (+8 pp).
  • Failure modes: Remaining failures cluster into pipeline bugs, train/validation domain gap, and model-capacity walls.Examples include under-abstraction, discovery-script loss, silent bypass, schema drift, and hardware-limited speaker diarization.

5 CONCLUSION AND LIMITATIONS

SkillEvolver reframes skill creation as artifact-level adaptation through deployed use, while its evaluation remains bounded by single-task, benchmark, process-signal, and model-coverage limitations.

  • Conclusion: SkillEvolver acquires reusable procedural skills from bounded deployment-time trials without updating model weights.Its loop combines strategy-diversified exploration, contrastive artifact edits, and independent auditing.
  • Limitations: The headline evaluation reflects a single configuration, Claude Opus 4.6 + Claude Code, because alternative-LLM benchmark sweeps were not run due to cost.Cross-LLM parity is presented as a design property rather than a benchmark-scale empirical finding.
  • Limitations: R=2 is a compute-budget choice rather than a measured optimum, and deeper refinement sweeps were not run at scale.The R=1→R=2 ablation attributes roughly two-thirds of the gain over the curated baseline to the second iteration.
  • Limitations: Evaluation covers 83 binary SkillsBench tasks and only three KernelBench tasks with scalar speedup.This reflects the youth of skill-authoring evaluation.
  • Limitations: Richer process signals such as step-level grounding, intermediate verifier checks, and critical-path latency remain uncharacterized.The agent currently consumes per-trial tokens, turns, and wall-clock information when contrasting traces.
  • Limitations: SkillEvolver addresses one newly arrived task at a time and does not organize or maintain a population of artifacts.Cross-task reuse, library deduplication, and parent–sibling specialization remain open.

A.1 SKILLEVOLVER FULL PSEUDOCODE

Algorithm 1 specifies SkillEvolver’s strategy-diversified, deploy-then-refine pipeline, including explicit trial routing and live-skill deployment.

  • A.1 SKILLEVOLVER FULL PSEUDOCODE: The pipeline uses strategy-diversified sampling and deploys the current candidate skill as a real dependency in trial containers.Algorithm 1 makes both design choices explicit.
  • A.1 SKILLEVOLVER FULL PSEUDOCODE: The pseudocode routes parallel trials through explicit strategy files and a per-trial environment index.At later iterations, strategies target observed weak spots of the current candidate skill.

A.2 AUDITOR CHECK LIST

The Auditor applies nine mechanical checks covering both standard content leakage and failures specific to deployed-skill use.

  • A.2 AUDITOR CHECK LIST: Checks 1–6 target standard content-level leakage patterns, while Checks 7–9 target deployment-specific failures.The latter include parametric-axis under-abstraction, primary-action hoisting, and silent-bypass.
  • A.2 AUDITOR CHECK LIST: Deployment-specific checks are possible because refinement uses deployed-skill handoff traces rather than the authoring agent’s self-reflection.This grounds auditing in how another agent actually receives and uses the skill.

A.3 CONTAMINATION CONTROLS

Contamination controls combine held-out validation, workspace isolation, strategy-diversified deployment, surgical refinement, auditing, and final validation.

  • A.3 CONTAMINATION CONTROLS: All evolution iterations run on Ttrain, while validation runs on Tval with different data, filenames, and sometimes sub-domain.The curated training skill is deleted before exploration so it is never reachable.
  • A.3 CONTAMINATION CONTROLS: Each iteration writes K strong-prior strategies, routes them to parallel trials, and deploys the candidate skill as a live dependency.After the first iteration, strategies target the candidate’s observed weak spots.
  • A.3 CONTAMINATION CONTROLS: At bootstrap, explicit strategy files control rollouts even though no domain skill has yet been learned.The minimal skill assigns rollout i to strategy file s0,i.
  • A.3 CONTAMINATION CONTROLS: Contrastive refinement compares winning and losing traces, then applies a surgical patch rather than rewriting the candidate.For later iterations, the contrast asks where the deployed skill misled, underspecified, or failed to guide the agent.
  • A.3 CONTAMINATION CONTROLS: A revised candidate is accepted only when the Auditor is clean and at least 3K/4 trial checks pass.The final artifact is validated on held-out Tval.
  • A.3 CONTAMINATION CONTROLS: A workspace whitelist denies tool calls outside a per-run prefix and blocks traversal or paths resolving into the curated training-skill slot.Validation directories, curated validation skills, and test suites remain unreachable.

A.4 SKILLCREATOR-SKILLSBENCH (CONTROL BASELINE) ALGORITHM AND ALIGNMENT TABLE

SkillCreator-SkillsBench is a control baseline that replaces the original human roles with isolated subagents while retaining the Improver in the main session. Its design is aligned with SkillEvolver and the non-refining ablation for controlled comparison.

  • SkillCreator-SkillsBench adapts Anthropic’s skill-creator by replacing human roles with Eval Designer, Grader, and Analyzer subagents.The Improver remains in the main session.
  • The control baseline is compared with SkillEvolver at R=1 and R=2 across trial counts, budgets, oracle policy, isolation, and authoring mechanism.
  • The adaptation preserves a structured skill-authoring pipeline while removing direct human participation from evaluation and feedback.

A.5 SKILLSBENCH TASK LIST AND CATEGORIZATION

The evaluation uses an 83-task SkillsBench scope spanning more than 15 professional domains, with symmetric exclusions for paid APIs and unstable infrastructure. The comparison also tracks deployment-specific auditing and shared anticheating controls across pipelines.

  • Task scope: The runnable SkillsBench scope contains 83 tasks spanning 15+ professional domains, including web development, data science, DevOps, chemistry, finance, and security.
  • Task scope: Four tasks are excluded symmetrically because two require paid external APIs and two have persistent infrastructure instability.The exclusions leave the 83-task paper scope.
  • Evaluation controls: The Auditor covers content-level leakage and deployed-skill failures such as under-abstracted parameters, unhoisted primary actions, and silent bypass.Checks 7–9 are observable because refinement uses traces from the candidate skill as a live dependency.
  • Baseline alignment: SkillCreator-SkillsBench uses isolated subagents for evaluation and feedback, while its algorithm performs local A/B self-tests before Harbor validation.
  • Baseline alignment: SkillEvolver, its R=1 ablation, and SkillCreator-SkillsBench share train/test splitting and workspace whitelisting as a two-layer anticheating design.

A.6 TRAINING-VARIANT GENERATION METHODOLOGY

Training variants preserve task structure while resampling inputs and remain disjoint from validation data, enabling evaluation of reusable skill behavior rather than direct instance reuse. Case studies show both successful transfer and failure from domain-specific overfitting, while KernelBench tests optimization transfer across architectures.

  • Training-variant generation: Training variants are disjoint from validation on filenames and data values, while preserving task structure, output shapes, and test-specification formats.The curated training skill is deleted at source and is never reachable during exploration.
  • Evaluation records: Per-task results record Pass@5, finalization choices, phase-level performance, and authoring cost across the evaluated conditions.
  • SkillsBench cases: 0.2 → 1.0 at R=2 on manufacturing-fjsp-optimization followed refinement that promoted the primary script to the top of the skill.The refinement addressed trials that stalled on which script to invoke first.
  • SkillsBench cases: 0.2 → 1.0 at R=2 on paper-anonymizer followed preservation of a missing discovery helper required to execute the prescribed strategy.The fix generalized into Auditor Check 9.
  • SkillsBench cases: 0.0 → 1.0 at R=2 on virtualhome-agent-planning followed a description-level change that raised Skill-tool invocation to 5/5.The skill body was unchanged; the description named the task and the relevant parser failure.
  • SkillsBench limitations: Single-domain training can encode a rule that fails under domain shift, as court-form-filling remained at 0/5 after refinement despite 43/47 verifier sub-tests passing.The case motivates multi-domain training because refinement received no signal to abstract away the medical-form heuristic.
  • KernelBench transfer: 1.326 → 2.226 was the largest KernelBench gain, while evolved skills improved mean reward on deepnarrowmlp, shufflenet, and gru.The reported pattern is architecture-aware optimization procedure rather than memorization of one kernel trick.
  • KernelBench limitations: KernelBench provides limited secondary evidence because it has only three tasks, uses a different metric, and shows less stable R=1 versus R=2 behavior than SkillsBench.
Loading 2605.10500v1…