Source-linked AI summary
SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents
Ziao Zhang, Kou Shi, Shiting Huang, Avery Nie, Yu Zeng, Yiming Zhao, Zhen Fang, Qishen Su, Haibo Qiu, Wei Yang, Qingnan Ren, Shun Zou, Wenxuan Huang, Lin Chen, Zehui Chen, Feng Zhao
TL;DR
Existing benchmarks leave open whether agents can discover, repair, and coherently maintain skills from experience. SkillFlow evaluates this with a DAEF-structured benchmark and sequential lifelong-learning protocol, finding selective rather than universal gains, including a +8.43-point improvement for Claude Opus 4.6.
Problem
Existing benchmarks test using provided skills but leave open whether agents can discover skills from experience, repair failures, and maintain a coherent library over time.
Method
SkillFlow provides 166 tasks across 20 families organized by DAEF and evaluates agents sequentially from an empty library using trajectory- and rubric-driven skill patches.
Results
Claude Opus 4.6 improves task success from 62.65% to 71.08% (+8.43 points), while other models show limited or negative gains despite skill use.
Takeaways & Limitations
Lifelong skill evolution yields selective rather than universal gains, with stronger stacks converting experience into compact reusable procedures and weaker ones showing fragmented, unstable skill libraries.
Takeaways & Limitations
The protocol evaluates lifelong learning within a single task family, avoiding heterogeneous workflow interleaving and its retrieval confounds.
Abstract
from arXiv · showhide
As the capability frontier of autonomous agents continues to expand, they are increasingly able to complete specialized tasks through plug-and-play external skills. Yet current benchmarks mostly test whether models can use provided skills, leaving open whether they can discover skills from experience, repair them after failure, and maintain a coherent library over time. We introduce SkillFlow, a benchmark of 166 tasks across 20 families in which task construction within each family follows a Domain-Agnostic Execution Flow (DAEF) that defines an agent workflow framework, allowing these tasks to share a consistent workflow. Agents are evaluated under an Agentic Lifelong Learning protocol in which they begin without skills, solve tasks sequentially within each family, externalize lessons through trajectory- and rubric-driven skill patches, and carry the updated library forward. Experiments reveal a substantial capability gap. For Claude Opus 4.6, lifelong skill evolution improves task success from 62.65% to 71.08% (+8.43 points). However, high skill usage does not necessarily imply high utility: Kimi K2.5 gains only +0.60 points despite 66.87% skill usage, while Qwen-Coder-Next reaches only a 44.58% task completion rate and still regresses relative to the vanilla setting. SkillFlow contributes a structured testbed for this direction and an in-depth empirical analysis of skill discovery, patching, transfer, and their failure modes under lifelong evaluation.
1 Introduction
SKILLFLOW addresses whether autonomous agents can discover, repair, and maintain reusable skills across sequential tasks, rather than merely use provided skills. It introduces a structured benchmark and lifelong protocol, with experiments showing uneven benefits across models.
- Evaluation protocol: Agents begin without skills, externalize experience into reusable artifacts, revise them through patches, and transfer them across tasks sharing a common workflow.This lifelong setting contrasts with conventional static-skill evaluation.
- Motivation: SKILLFLOW benchmarks whether agents can extract reusable skills from experience, repair them after failures, and maintain an evolving library across sequential tasks.The benchmark contains 166 runnable tasks across 20 task families.
- Benchmark design: The benchmark uses DAEF to organize tasks around shared operational workflows and support controlled cross-domain task construction.A dual-agent pipeline generates and reviews task families before human verification.
- Findings: Claude Opus 4.6 improves task success from 62.65% to 71.08% (+8.43 points) under lifelong skill evolution.The result is presented as evidence of a substantial capability gap across evaluated models.
- Findings: High skill usage does not guarantee utility: Kimi K2.5 gains only +0.60 points at 66.87% skill usage, while Qwen-Coder-Next reaches 44.58% completion and regresses versus vanilla.The benchmark also analyzes failure patterns including fragmented growth, erroneous logic reinforcement, and unreliable repair.
2 SKILLFLOW
SKILLFLOW structures lifelong skill evaluation around shared, domain-agnostic workflows and sequential task families. Its protocol constructs and updates an external skill library from task trajectories and verifier feedback, enabling evaluation of skill maintenance, repair, and improvement over time.
- Benchmark scope: SKILLFLOW spans 20 workflow families and 166 tasks across five broad domains, with family-local sequences, fixed difficulty orders, and associated verifier styles.The benchmark covers realistic workplace workflows including spreadsheet planning, OCR and PDF extraction, document editing, compliance analysis, and quantitative tasks.
- Domain-Agnostic Execution Flow: DAEF abstracts shared workflow skeletons by removing domain-specific entities, file names, and business semantics while preserving operation types and dependency structure.Task instances are represented as domain-grounded workflow graphs, and DAEFs retain the stable operational topology for task-family construction and generalization.
- Cross-domain structure: DAEF correspondence across domains supports systematic evaluation of cross-domain skill transfer by instantiating distinct tasks from the same abstract workflow.The benchmark also organizes five top-level categories with corresponding DAEF names.
- Agentic Lifelong Learning Protocol: The lifelong protocol orders tasks within each family, starts the first task without skills, and generates skill patches from execution trajectories and verifier-derived rubrics.For later tasks, the agent uses the current library and incrementally applies patches that may add, revise, or delete skills.
- Evaluation objective: SKILLFLOW evaluates whether agents can continuously maintain, repair, and improve a reusable skill repository rather than merely benefit from static external memory.The protocol carries the evolving library across sequential tasks and preserves its update history through incremental patches.
3 Experiments
Experiments evaluate lifelong skill evolution across 11 model variants and compare completion, efficiency, skill generation, and reuse under vanilla and evolving-library settings. Results show highly variable transfer: repaired, compact libraries can improve performance, while incorrect or proliferating skills can cause regressions and persistent downstream drift.
- Experimental Setup: 11 model variants are evaluated with matched agent harnesses under vanilla execution and lifelong skill evolution.The benchmark reports task completion, turns, cost, output tokens, cumulative skills, and skill-use rates.
- Main Results: Claude Opus 4.6 improves mean task success from 62.65% to 71.08% (+8.43 points), the clearest positive result in the main table.It completes 118 of 166 tasks under skill evolution versus 104 under vanilla execution.
- Main Results: Skill evolution varies sharply across settings: GPT 5.3 Codex declines from 52.41% to 46.39% (-6.02 points), while Claude Sonnet 4.6 remains at 56.63%.Other models show positive, modest, or negative transfer, demonstrating that skill evolution is not uniformly beneficial.
- Efficiency and Transfer: Skill evolution can improve completion at comparable or lower cost, but some settings incur higher cost with limited or no gains.Figure 5 presents these settings as a completion–cost Pareto frontier.
- Efficiency and Transfer: Gains span benchmark groups, with more negative gains in Finance & Economics and more positive transfer in Data & Document Intelligence.The domain-grouped analysis does not show improvement concentrated in a single category.
- Failure Modes and Library Structure: Positive transfer aligns more closely with repairing incorrect skills and maintaining compact, reusable libraries than with skill volume or usage alone.Incorrect skills can induce persistent downstream drift, whereas fragmented skill inflation often produces weak or negative gains.
4 Related Work
Related work studies tool-using agents, reusable procedural skills, and experience-driven skill evolution. However, existing efforts provide limited evaluation of autonomous skill derivation and cross-task transfer.
- Controlled agent benchmarks examine tool use, multi-step reasoning, tool-call error correction, and inefficiency in complex tool-integrated reasoning.
- Recent skill research treats skills as reusable procedural knowledge connecting models and workflows, but emphasizes infrastructure and downstream performance.
- Across these lines of work, evaluation of skill derivation and cross-task transfer remains limited.
- Other methods discover or evolve skills from interaction by distilling patterns, incorporating feedback and failures, or accumulating long interaction traces.
5 Conclusion
SKILLFLOW benchmarks lifelong skill discovery and evolution through sequential workflow-level evaluation. Its findings indicate selective gains: stronger agent–model stacks consolidate experience, while weaker ones exhibit unstable skill evolution.
- SKILLFLOW evaluates lifelong skill discovery and evolution using a dual-agent construction pipeline, sequential protocol, and Domain-Agnostic Execution Flow.
- The benchmark tests skill transfer at the workflow level rather than through superficial lexical overlap.
- Lifelong skill evolution produces selective rather than universal gains across agent–model stacks.
- Stronger stacks consolidate externalized experience into compact reusable procedures, whereas weaker stacks show fragmented libraries and unstable reinforcement under feedback.
- Continuous skill evolution is presented as a practical mechanism for acquiring and consolidating knowledge across domains in lifelong learning systems.
A.1 Comparison
Table 2 compares SKILLFLOW with representative benchmarks across dimensions of the skill lifecycle. The comparison covers whether benchmarks evaluate skill impact, generation, revision, lifelong reuse, transfer, trajectory grounding, and usage utility.
- Table 2 organizes benchmark comparison around seven skill-lifecycle dimensions, including explicit skill evaluation, self-generation, revision, and lifelong accumulation.
- The comparison also evaluates cross-scenario procedural transfer and whether skills are grounded in interaction trajectories.
- Usage evaluation measures alignment between skill utility and actual skill usage.
A.2 Benchmark Statistics
The benchmark contains 20 workflow families across five broad domains, with 166 tasks organized by formal family names and family-local groupings. Family sizes range from 8 to 9 tasks.
- The benchmark spans five domains: Finance & Economics, Operations & Supply Chain, Healthcare & Life Sciences, Governance & Strategy, and Data & Document Intelligence.
- These domains contain 20 workflow families and 166 tasks in total.
- Family sizes range from 8 to 9 tasks.
- The workflow-family grouping uses formal family names rather than internal source identifiers.
- The listed families include financial analysis, supply-chain and production planning, healthcare analysis, auditing, compensation modeling, and document-related workflows.
A.4 Workflow Details
The benchmark organizes tasks into workflow families with shared DAEF descriptions, spanning analytical, planning, document, spreadsheet, presentation, and financial operations. Each workflow specifies sequential processing steps from reading inputs through structured outputs.
- Workflow-family organization: Table 3 catalogs workflow families with their workflow definitions, seed-benchmark lineage, task summaries, and corresponding DAEF descriptions.The Source column identifies whether a family derives from SkillsBench or GDPval.
- Analytical and financial workflows: Financial workflows normalize and align records before computing aggregates, comparisons, correlations, or rolling balances for structured analysis.Examples include industry correlation, financial statement rolling, and SEC 13F analysis.
- Planning workflows: Planning workflows read constraints and state, compute candidate plans or demand updates, compare resource or capacity outcomes, and output workbooks or plan sheets.The families cover replenishment, inventory-finance integration, operational recovery, and production capacity planning.
- Document and data workflows: Document and data-processing workflows extract, normalize, validate, and transform heterogeneous records into repaired, standardized, reconciled, or analyzed outputs.These include medical standardization, auditing, fraud detection, embedded data repair, OCR extraction, HWPX automation, and cross-format reconciliation.
- Presentation and reporting workflows: Presentation and structured-report workflows update formatting, compute pivot indicators, compare grouped summaries, and generate completed or formatted documents.The task families include PPT formatting optimization and sales pivot analysis.
B.3 Skill Library Growth and Composition
Figures 7 and 8 characterize how evolved skill libraries change during sequential evaluation. They jointly show library-size growth and the file-kind composition of stored artifacts.
- Skill count growth: Figure 7 tracks the cumulative number of stored skills as tasks progress, revealing whether models consolidate experience or keep expanding their libraries.This view supports comparison of library growth patterns across models and task sequences.
- Skill file-kind composition: Figure 8 summarizes the file-kind composition of evolved skill libraries, distinguishing procedural summaries, auxiliary files, and fragmented artifact collections.The composition view complements skill-count growth by describing what the libraries contain.
C.1 Metric Notes
The evaluation reports task completion alongside interaction efficiency, output volume, and skill-generation or reuse statistics. Complementary figures and controls help distinguish skill-evolution effects from raw-history access and inspect family-level variation.
- Skill-library metrics: Final #Skills measures cumulative skills generated for a task family, while %use measures tasks reading or calling at least one previously stored skill.These statistics are reported because they are consistently available across the systems in Table 1.
- Efficiency metrics: Completion–interaction-turns and completion–output-tokens Pareto views compare task success with interaction length and generated output volume.The output-token view is useful for diagnosing verbosity-driven gains or regressions.
- Control condition: The history-context control tests whether full prior interaction histories explain protocol gains without externalizing experience into a skill library.It compares vanilla, history-context, and full-protocol settings for Claude Opus 4.6.
- Family-level analysis: Figure 11 resolves the benchmark into individual workflow families to inspect localized gains, negligible changes, and regressions beyond coarse domain-grouped results.The family-level heatmap provides a more granular complement to the main-text heatmap.
D.1 Skill Evolution Patch
The skill patch develops a reusable Excel-formula workflow centered on reading the verifier, writing formulas with openpyxl, and making cached values available when required. It also records failure-prevention rules for formula evaluation, XML patching, layout mapping, and verification.
- Verifier-driven workflow: Read the test or verifier first to determine whether it checks formula strings, computed values, cached values, addresses, tolerances, or workbook structure.The verifier may use data_only=True, which requires cached values rather than merely formula strings.
- Formula values: openpyxl does not evaluate formulas, so verifiers reading data_only=True may receive None unless cached values are written separately.The decision rule depends on whether verification checks numeric values or formula text.
- Cached-value patching: Use XML-level cached-value patching as the preferred fallback when external recalculation is unavailable or unreliable.The recommended workflow computes expected values in Python and patches worksheet <v> elements after saving the workbook.
- Failure prevention: Run XML patching once and handle both numeric and string formula results to avoid corrupting formula cells.Multiple patches can corrupt the XLSX structure, while complex or unrecognized formulas may require task-specific computation logic.
- Workbook construction: Write formulas with correct absolute and relative references after mapping sheets, target cells, source ranges, labels, and existing formulas precisely.The workflow emphasizes preserving workbook formatting and avoiding off-by-one row or column errors.
- Validation: Self-verify with pytest and preserve all existing formatting, values, and structure while modifying only the target range.The guidance also rejects hardcoded formula data and warns against declaring success without testing.