EvoSkill: Automated Skill Discovery for Multi-Agent Systems

Salaheddin Alzubi; Noah Provenzano; Jaydon Bingham; Weiyuan Chen; Tu Vu

Source-linked AI summary

EvoSkill: Automated Skill Discovery for Multi-Agent Systems

Salaheddin Alzubi, Noah Provenzano, Jaydon Bingham, Weiyuan Chen, Tu Vu

arXiv:2603.02766v1Mar 3, 2026cs.AIcs.MA

TL;DR

Coding agents remain weak on specialized tasks because flexibility does not supply domain expertise, while existing skills are largely hand-crafted or tied to low-level artifacts. EvoSkill instead discovers and refines reusable skills through failure analysis, retaining candidates on held-out validation while freezing the model. It improves OfficeQA and SealQA and transfers a SealQA-evolved skill zero-shot to BrowseComp.

Problem
Coding agents’ generality does not confer specialized domain expertise, and most agent skills require costly hand-crafting while existing evolutionary methods optimize model- and task-coupled prompts or code.
Method
EvoSkill analyzes execution failures, proposes new skills or edits, materializes structured skill folders, and selects improving programs on held-out validation while the underlying model remains frozen.
Results
EvoSkill improves performance across OfficeQA and SealQA and transfers a SealQA-evolved skill zero-shot to BrowseComp, improving accuracy by 5.3% without modification.
Takeaways & Limitations
Skill-level optimization produces structured, interpretable capabilities that accumulate across iterations and can transfer beyond the training task.
Takeaways & Limitations
The evaluation uses a single run per configuration, leaving variance analysis across multiple seeds for future work.

Abstract

from arXiv · show

Coding agents are increasingly used as general-purpose problem solvers, but their flexibility does not by itself confer the domain expertise needed for specialized tasks. Recent work addresses this through \textit{agent skills}: reusable workflows, and code, that augment agents with domain-specific capabilities. Most skills today are hand-crafted, and existing evolutionary approaches optimize low-level artifacts (e.g. prompts \& code) that are tightly coupled to specific models and tasks. We introduce \textbf{EvoSkill}, a self-evolving framework that automatically discovers and refines agent skills through iterative failure analysis. EvoSkill analyzes execution failures, proposes new skills or edits to existing ones, and materializes them into structured, reusable skill folders. A Pareto frontier of agent programs governs selection, retaining only skills that improve held-out validation performance while the underlying model remains frozen. We evaluate EvoSkill on two benchmarks: OfficeQA, a grounded reasoning benchmark over U.S.\ Treasury data, where it improves exact-match accuracy by \textbf{7.3\%} (60.6\% $\to$ 67.9\%); and SealQA, a search-augmented QA benchmark with noisy retrieval, where it yields a \textbf{12.1\%} gain (26.6\% $\to$ 38.7\%). We also investigate the zero-shot transfer capabilties of skills evolved on one task to the other; in particular: skills evolved from SealQA transfers zero-shot to BrowseComp, improving accuracy by \textbf{5.3\%} without modification demonstrating that skill-level optimization produces transferable capabilities beyond the training task.

1 Introduction

EvoSkill addresses the limited domain expertise of flexible coding agents by automatically discovering reusable skills through failure analysis, improving performance across benchmarks and transferring to an unseen task.

Coding agents can solve tasks across domains, but flexibility alone does not provide the domain expertise needed for consistently strong specialized-task performance.
Agent skills augment general-purpose coding agents with reusable, domain-specific workflows, instructions, and supporting code, but most are hand-crafted and costly to scale.
EvoSkill discovers and refines agent skills through failure-driven textual feedback rather than directly optimizing prompts or codebases.
7.3% exact-match accuracy gain on OfficeQA raised performance from 60.6% to 67.9%, while SealQA improved by 12.1% from 26.6% to 38.7%.
A skill evolved on SealQA transferred zero-shot to BrowseComp, improving accuracy by 5.3% without modification.

2 Methodology

EvoSkill evolves skills within a frozen-model agent harness by diagnosing failures, building candidate skill folders, and selecting programs through held-out validation on a bounded frontier.

EvoSkill keeps the underlying model and base code fixed while evolving only skill repositories and agent metadata.
The framework uses Executor, Proposer, and Skill-Builder agents to run tasks, diagnose failures, and materialize proposals into concrete skill folders.
The Proposer uses execution traces, predicted answers, ground-truth answers, and feedback history to identify capability gaps and propose new skills or edits.
Candidate programs are evaluated on a held-out validation set and enter the fixed-capacity frontier only when they outperform its weakest member.
Parents are selected round-robin from the frontier, while training batches expose failures that guide subsequent skill mutations.
The dataset is stratified into training, validation, and test subsets, with the test set never exposed during evolution and reserved for final evaluation.

3 Experiments

EvoSkill is evaluated across challenging grounded-reasoning settings, training configurations, and qualitative skill discoveries. On OfficeQA, skill merging produces the strongest result, while discovered skills target concrete extraction and quantitative-analysis failures.

Experimental Scope: The experiments assess iterative skill evolution, training-setup effects, zero-shot transfer, and qualitative properties of discovered capabilities.The evaluation includes benchmark performance, training conditions, transfer, and representative skill examples.
3.1.1 Benchmark: OfficeQA uses Treasury Bulletins and requires cross-document retrieval, dense-table navigation, and basic quantitative reasoning.The corpus contains approximately 89,000 pages, and questions require synthesizing information across an average of two documents.
3.1.3 Results: On exact match, 5% training improves accuracy from 60.6% to 63.4% (+2.8%), while 10% training reaches 65.8% (+5.2%).The 15% split reaches 64.5%, slightly below the 10% run, suggesting diminishing returns or mild overfitting as training data grows.
3.1.3 Results: The skill-merge configuration achieves 67.9% exact match, a +7.3% improvement over baseline and the strongest result among individual runs and merged configurations.The improvement pattern remains consistent across tolerance levels, with gains ranging from 2.7-4.5% at stricter tolerances.
3.1.4 Qualitative Analysis of Discovered Skills: EvoSkill discovers interpretable skills addressing specific failure modes, including adjacent-cell misreads, wrong metric selection, and time-granularity errors.A quantitative-analysis skill adds validation checkpoints for transformations, date alignment, and sample-versus-population statistics.
3.1.4 Qualitative Analysis of Discovered Skills: Each configuration was evaluated in a single run because Opus 4.5 made repeated evolution costly, leaving variance analysis across seeds for future work.This limits direct assessment of run-to-run variability.

3.2 SealQA

SealQA tests search-augmented question answering under conflicting and noisy web retrieval. EvoSkill improves accuracy by enforcing exhaustive search and source verification, and the resulting skill transfers to BrowseComp without modification.

3.2.1 Benchmark: SealQA evaluates fact-seeking agents when web search returns conflicting, noisy, or unhelpful results.Unlike OfficeQA’s fixed corpus, SealQA requires navigating the open web under adversarial retrieval conditions.
3.2.2 Setup: The SealQA evaluation uses 111 questions, a 10% training split, held-out test questions, and 1.5 evolution epochs.Frontier selection uses the remaining partition methodology described for the benchmark setup.
3.2.3 Results: 12.1% absolute gain: EvoSkill improves SealQA accuracy from 26.6% to 38.7%.The discovered search-persistence-protocol requires term expansion, multi-source verification, and completeness checks before committing to an answer.
3.3 Zero-Shot Skill Transfer: The search-persistence-protocol transfers zero-shot to BrowseComp without edits, improving accuracy from 43.5% to 48.8% (+5.3%).The transferred skill targets exhaustive search before answer commitment across a different fact-seeking benchmark.

4 Related Work

Prior work develops reusable agent skills, textual-feedback optimization, and evolutionary search, while EvoSkill evolves structured skill modules designed to persist and transfer across tasks.

4.1 Agent Skills: Reusable agent skills augment general-purpose agents with modular capabilities, building on work in embodied AI and software engineering.Voyager uses executable skill libraries, while the Agent Skills specification defines portable folders with metadata, instructions, scripts, and reference materials.
4.2 Textual Feedback and Evolutionary Optimization: Textual feedback can iteratively improve generated artifacts, but Self-Refine operates on individual outputs and does not accumulate knowledge across iterations.Feedback Descent addresses accumulation by maintaining a frontier of candidates and feedback history.
4.2 Textual Feedback and Evolutionary Optimization: Evolutionary methods such as AlphaEvolve and GEPA optimize codebases or prompts through iterative search and Pareto-based selection.These approaches operate on lower-level artifacts than EvoSkill.
4.3 Transfer Learning in LLM Agents: EvoSkill evolves structured, reusable capability modules rather than prompts or code, making them interpretable, composable, and transferable without modification.Its transfer design uses self-contained folders with explicit trigger conditions and procedural instructions.
4.3 Transfer Learning in LLM Agents: A skill evolved on SealQA transfers zero-shot to BrowseComp, improving accuracy by 5.3 percentage points without modification.The authors note that broader investigation across more diverse task pairs is still needed.

5 Conclusion

EvoSkill automatically discovers and refines reusable skills through iterative failure analysis, improving two benchmarks and providing evidence of zero-shot transfer. The authors identify broader domain, multimodal, library, and transfer evaluation as future directions.

Conclusion: EvoSkill automatically discovers and refines reusable agent skills through iterative failure analysis at the skill level.The resulting capabilities accumulate over iterations and are structured and interpretable.
Conclusion: +7.3% on OfficeQA and +12.1% on SealQA demonstrate improvements across two distinct benchmarks using only small training subsets.The experiments also provide direct evidence of zero-shot transfer from SealQA to BrowseComp by +5.3%.
Future Work: Future work should evaluate broader domains, multimodal tasks, shared skill libraries, and transfer across tasks, models, and agent harnesses.The authors specifically distinguish potentially domain-general skills from domain-specific skills.

A EvoSkill Generated Skills

This section presents examples of skills generated by EvoSkill for different agentic tasks.

Generated Skills: The appendix presents examples of skills generated using EvoSkill on different agentic tasks.

A.1 OfficeQA Skills

The OfficeQA example skill standardizes economic time-series analysis by structuring nominal and CPI data, adjusting values for inflation, running linear regression, and formatting results.

Skill Structure: The economic-timeseries-analysis skill combines a SKILL.md file with Python scripts for multi-step economic data analysis.It targets tasks involving CPI values, inflation adjustment, linear regression, and nominal dollar values.
Step 1: Data Collection: The workflow gathers nominal values, CPI values, and a base period using consistent YYYY-MM period formatting.The inputs are represented as period-value records alongside the selected base period.
Step 2: Inflation Adjustment: Real Value = Nominal Value x (CPI_base / CPI_current) defines the inflation-adjustment step.For April 1970, the example converts $318.44 using CPI_base 38.8 and CPI_current 39.0 to $316.81.
Step 3: Linear Regression: Linear regression uses period indices 0, 1, 2, ... as x-values and inflation-adjusted real values as y-values.The workflow invokes an analysis script and returns slope and intercept values.
Step 4: Format Output: Regression output is formatted as [slope, intercept] rounded to two decimal places.The example result is [44.00, 231.52].
Validation: The implementation validates required fields and rejects empty inputs or missing CPI values before analysis.It also requires at least two data points for regression.

A.2 SealQA

The search-persistence-protocol skill structures exhaustive web research for ambiguous or factual questions, requiring interpretation expansion, repeated searching, verification, and completeness checks.

The protocol combines expansion, search, verification, completeness, derivation, and delayed conclusion into a single workflow.Its stated purpose is preventing premature search termination.
The protocol expands ambiguous terms into all reasonable interpretations before searching each interpretation separately.It explicitly warns against assuming the first interpretation is correct.
It requires checking at least three independent sources before concluding a factual answer.If fewer sources are found, the agent must reformulate queries and continue searching.
Before reporting inability to find data, the agent tries multiple query formulations, related searches, and derivation from related data.
When a specific database, API, indicator, or dataset is identified, the agent must attempt to fetch it before concluding that data is unavailable.The protocol rejects merely telling users they would need to query an accessible source.
Enumeration questions receive additional completeness checks through complete-list searches, cross-referencing, and investigation of discrepant counts.

B Agent Prompts

The agent prompts assign failure diagnosis and skill design to a Proposer, then direct a Skill-Builder to implement validated, reusable skills under explicit structural and maintenance requirements.

B Agent Prompts: The B Agent Prompts appendix provides role-specific prompts for agents that propose and build skills.
B.1 Proposer: The Proposer analyzes execution traces, predicted answers, and ground truth to diagnose failures and propose targeted skill improvements.Ground-truth answers support root-cause diagnosis but are not propagated into generated skills.
B.2 Skill-Builder: The Skill-Builder implements complete skills using prescribed structure, SDK integration, documentation, maintainability, validation, and edge-case handling.It must first follow the skill-creator guidance before implementation.
B.1 Proposer: Before proposing a skill, the Proposer brainstorms alternatives, inventories existing skills, reviews feedback history, and decides between creating or editing a skill.The process is intended to avoid redundant proposals and incorporate prior successes or regressions.
B.1 Proposer: The Proposer identifies the needed capability, inputs, outputs, integration, justification, and relevant prior iterations.
B.1 Proposer: The Proposer should favor broad, reusable capabilities and avoid narrow, overlapping, or redundant skills.Skills are proposed for reusable workflows, multi-step procedures, output structures, or missing access and computational capabilities.
B.1 Proposer: Examples distinguish editing an existing financial skill for multi-period calculations from creating a new Treasury bond-notation parser when no skill covers the gap.The bond example addresses 32nds notation, suffix handling, validation, and conversion examples.

C Scoring & Data setup

The evaluation uses deterministic fuzzy matching that extracts and normalizes numerical or textual content, with exact numerical agreement required for primary evaluation.

For numeric answers, it extracts values with local context, normalizes units, and compares base values using a relative tolerance.Primary evaluation sets τ = 0, requiring exact numerical agreement.
The scorer filters prediction numbers from 1900–2100 to reduce spurious matches against incidental year references.This filter is relaxed when the ground truth itself is a year or contains significant non-numeric text.
Hybrid answers require both constituent numeric values and key textual elements to be recovered in the prediction.Text matching uses case-insensitive normalized substring containment.
The training loop also uses a weighted multi-tolerance score across τ values from 0.0 to 0.10, favoring stricter thresholds.

D Environment Branches

Agent configurations are managed as git-backed programs whose parent–child lineage supports frontier-based self-improvement and targeted skill mutations.

Each program is stored on a dedicated git branch with YAML metadata for lineage, generation, prompts, tools, and evaluation scores.A base program starts at generation 0 with no parent.
At each iteration, the highest-scoring frontier program is selected as parent and mutated into a child with a targeted skill or prompt modification.Skill-only mode changes a skill file, while prompt-only mode rewrites the system prompt.