Source-linked AI summary
ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?
Tianyi Guan, Yiding Wang, Haotong Yang, Siyuan Cao, Shirui Liu, Yi Hu, Jiaqi Li, Muhan Zhang
TL;DR
It is unclear whether LLM agents can autonomously evolve skills from task feedback and whether those skills improve later task performance. ContinualSkillBench evaluates this across structured task sequences, finding that sequential execution generally helps, while consistent consolidation into transferable skills remains difficult.
Problem
Whether agents can autonomously synthesize and evolve skills from task sequences and feedback remains systematically under-evaluated.
Method
ContinualSkillBench evaluates skill evolution across five domains, each containing 100 interconnected subtasks ordered by difficulty and opportunities for cross-task skill reuse.
Results
Sequential execution improves normalized reward in 14 of 15 model–domain combinations, with a 16.9% aggregate relative gain; in-context learning matches explicit skill maintenance on average.
Takeaways & Limitations
Continual interaction supports adaptation, but agents do not consistently consolidate experience into robust, transferable skills.
Takeaways & Limitations
The benchmark uses tasks curated from fixed sources and does not fully cover rare edge cases, shifted distributions, or substantially different instructions.
Abstract
from arXiv · showhide
Modern agent frameworks equip large language models with external skill libraries to solve complex tasks. However, it remains unclear whether these systems can effectively evolve their skills and whether the resulting skills improve task-solving capabilities. To bridge this gap, we introduce ContinualSkillBench, a dynamic evaluation framework for in-context continual skill learning. It covers five representative domains, each containing 100 interconnected subtasks ordered by increasing difficulty and opportunities for cross-task skill reuse. Our experiments show that sequential execution generally improves performance, but the gains vary substantially across models and domains. Moreover, in-context learning performs comparably to explicit skill maintenance on average, suggesting that much of the improvement arises from adaptation to prior context and feedback rather than reusable skill abstraction alone. Explicit skills nevertheless provide selective benefits for tasks requiring reusable procedures or precise outputs. We further find that less capable models tend to accumulate larger, more fragmented collections of task-specific skills. These findings show that current in-context skill evolution mechanisms can support continual adaptation, but still struggle to consistently consolidate experience into robust and transferable skills.
1 Introduction
ContinualSkillBench evaluates whether LLM agents can autonomously evolve skills through ordered task sequences with feedback, addressing the difficulty and expense of manually building comprehensive skill libraries. Across 15 model–domain combinations, sequential execution improves normalized reward in 14, with an aggregate relative gain of 16.9%, though gains vary substantially across models and domains.
- Motivation: Real-world tasks often exceed pretrained knowledge and processing logic, motivating agent skills that enhance task-solving performance during deployment.These skills are typically organized as structured documents and are used by mainstream agent platforms.
- Motivation: Agents must potentially synthesize and evolve skills autonomously from task descriptions and feedback because manually building comprehensive skill libraries is difficult and expensive.Prior work shows that well-written human-authored skills improve performance on corresponding tasks.
- Benchmark: ContinualSkillBench assesses continual skill evolution through logically progressive task sequences whose downstream objectives rely on skills practiced earlier.The framework is designed for continuous feedback learning rather than isolated tasks with fixed skill documents.
- Results: 16.9% aggregate relative gain: sequential execution improves normalized reward in 14 of 15 model–domain combinations, with substantial variation across models and domains.Sequential agents continuously update skills using task feedback, whereas independent baselines execute every task from scratch.
- Benchmark: Each of five domains contains 100 interconnected subtasks ordered by increasing difficulty and opportunities for cross-task skill reuse.The benchmark compares multiple model–harness configurations, including explicit skill maintenance, independent execution, and in-context learning baselines.
2 Related Work
Prior work equips agents with external tools or skills but often assumes static libraries, while parametric adaptation can cause catastrophic forgetting. Existing benchmarks largely assess fixed or curated skills, whereas ContinualSkillBench evaluates whether skills evolve, consolidate, and transfer across heterogeneous task streams.
- In-Context Skill Evolution for Agents: Agents increasingly use external tools or structured skills for complex domain-specific tasks, but static skill libraries limit adaptability in dynamic environments.Sequential adaptation through standard parametric updates can also cause catastrophic forgetting, motivating in-context continual learning.
- Benchmarks for Agent Skills and Continual Learning: Most general-purpose agent benchmarks evaluate task completion with fixed tools, instructions, or skills, while SkillsBench compares no-skill and curated-skill conditions without studying library evolution.This leaves skill evolution across sequential interactions insufficiently examined.
- Benchmarks for Agent Skills and Continual Learning: ContinualSkillBench studies skill evolution over long, heterogeneous task streams whose tasks are linked by recurring core skills rather than a single repeated task or fixed workflow.Its setting tests whether experience can be consolidated into structured skills, reused downstream, and examined across variation in this ability.
3 CONTINUALSKILLBENCH
ContinualSkillBench evaluates continual skill learning through five domain-specific streams of 100 progressively ordered subtasks, with feedback-driven skill updates available across tasks. Its curated sequences exhibit substantial skill recurrence and structured opportunities for cross-task transfer.
- Benchmark design: ContinualSkillBench contains five domain-specific task streams, each with 100 subtasks ordered by difficulty and potential for skill reuse.After each subtask, agents receive feedback and may create or modify repository skills for subsequent tasks.
- Benchmark design: Tasks are organized by core-skill dependencies, progressively moving from simpler to more challenging capabilities within each domain.The construction uses LLM-based skill labeling, dependency graphs, difficulty-constrained ordering, and human review.
- Transfer structure: 69.5% of eligible tasks reuse at least one core skill encountered earlier, while 35.5% of target-task core-skill requirements have a semantic counterpart in preceding history.Semantic counterparts are defined using cosine similarity of at least 0.85 between textual skill representations.
- Transfer structure: Curated ordering yields higher core-skill coverage than random permutations across all five domains and history windows of 1, 5, and 10 preceding tasks.This indicates that nearby tasks are more likely to require overlapping or semantically related core skills.
- Agent protocol: Each subtask follows instruction, execution, and reflection turns, with evaluator feedback guiding optional skill creation or revision in the repository.The benchmark supports multiple evaluation formats, including exact match, F1, numeric, rubric-based, and programmatic evaluation.
4 Experiments
Sequential execution generally improves performance across models and domains, but its gains vary by model, domain, and evaluator type. Pure in-context learning explains much of the aggregate benefit, while explicit skills provide selective advantages for reusable procedures and rigid outputs.
- Sequential performance: Sequential execution raises raw reward in 13 of 15 model–domain combinations and normalized reward in 14 of 15, with macro-average gains of +0.071 and +0.078.Relative improvements are 16.2% for raw reward and 16.9% for normalized reward.
- Model differences: GPT-5.3-Codex has the largest average normalized improvement (+0.098), followed by GPT-4o (+0.077) and Opus 4.7 (+0.058).The ordering of sequential gains does not follow Independent baseline performance.
- Domain differences: Healthcare shows the largest average normalized improvement (+0.149), while Finance, Law, Office, and Mathematics achieve +0.076, +0.058, +0.054, and +0.052.The only normalized-reward decrease is Opus 4.7 on Mathematics (−0.008).
- Evaluator effects: Evaluator effects vary: GPT-5.3-Codex gains +0.416 on Numeric and +0.091 on Exact Match tasks in Finance, versus +0.038 on Rubric tasks.Opus 4.7 instead gains +0.234 on Healthcare Rubric tasks but decreases by −0.192 on Mathematics Rubric tasks.
- In-context learning: Across Law, Finance, and Healthcare, average normalized rewards are 0.466 for Independent, 0.605 for ICL, and 0.602 for skill-maintaining Sequential execution.Sequential slightly exceeds ICL in Law and Finance but trails it in Healthcare, indicating no consistent aggregate advantage for explicit skill maintenance.
- Skill-library behavior: GPT-5.3-Codex forms a compact, frequently reused skill library, whereas GPT-4o accumulates a larger pool of fragmented, task-specific skills with limited reuse.The fragmented skills enlarge GPT-4o’s repository without proportional downstream utility and increase selection and maintenance burden.
5 Conclusion
ContinualSkillBench evaluates continual skill learning across long task sequences in five domains. Sequential execution generally improves performance, while in-context learning matches explicit skill maintenance on average and explicit skills help selectively.
- ContinualSkillBench evaluates continual skill learning across long task sequences in five domains.
- Sequential execution generally improves performance, but gains vary across models and domains.
- In-context learning performs comparably to explicit skill maintenance on average, while explicit skills selectively benefit reusable procedures or precise outputs.The improvement is attributed largely to adaptation to prior context and feedback.
Limitations · A Detailed Task Sources
The study is limited by curated task sources, restricted model and harness coverage, and incomplete representation of long-tail deployment conditions. It also documents a task suite combining foundational and domain-specific datasets across five core domains.
- Limitations: The benchmark’s tasks are curated from fixed sources, limiting coverage of rare edge cases, shifted distributions, and substantially different instructions.These long-tail deployment settings are more diverse and less controlled than the benchmark format.
- A Detailed Task Sources: The benchmark covers five core domains, with Table 3 summarizing the datasets used and their respective target capabilities.The supplied passage describes the datasets as the primary data sources integrated into CONTINUALSKILLBENCH.
- Limitations: Sequential evaluation covers only a limited set of representative models and agent harnesses because of high time and API costs.The study does not exhaustively evaluate additional model variants or agent environments.
- Limitations: The evaluation excludes additional Claude, GPT, and Gemini variants, as well as agent environments such as Cursor and Google CLI.The authors identify broader model and infrastructure coverage as future work.
- Limitations: Different models and agent frameworks may handle memory, skill retrieval, and execution control differently, motivating expanded coverage.This variation limits how broadly results from the evaluated configurations can be generalized.
- A Detailed Task Sources: CONTINUALSKILLBENCH integrates cross-domain foundational benchmarks with specialized domain-specific datasets to construct its evaluation suite.The task sources are presented as supporting a robust evaluation suite.
A.1 Dataset Descriptions · B Pairwise Dependency Evaluation and Graph-based Ordering
ContinualSkillBench combines cross-domain foundational benchmarks with specialized suites spanning Finance, Healthcare, Law, Math, and Office. These datasets target workflow execution, precision, reasoning, safety, document processing, and other domain-specific capabilities.
- A.1 Dataset Descriptions: Cross-domain benchmarks cover Finance, Healthcare, Law, and Office, evaluating multi-step workflows, tool manipulation, high-throughput text processing, and entity extraction.The benchmark integrates ClawBench, OneMillionBench, and GAIA for foundational general-assistant capabilities.
- A.1 Dataset Descriptions: Finance tasks assess high-precision numerical accuracy and strategic decision-making through hybrid table-text question answering, investment analysis, portfolio optimization, and financial knowledge.Sources include TAT-QA, InvestorBench, and FinBen.
- A.1 Dataset Descriptions: Healthcare tasks combine specialized medical knowledge, safe diagnostic reasoning, multi-agent consultation, and public-health scenarios to evaluate clinical safety and collective diagnostic accuracy.The benchmark adapts PhysicianBench, MedAgentsBench, and HealthBench.
- A.1 Dataset Descriptions: Law tasks emphasize textual precision, statutory interpretation, clause matching, structured reasoning, localized legal knowledge, advanced analysis, and citation matching.Sources include LegalBench, LawBench, harvey-labs, and Lexam.
- A.1 Dataset Descriptions: Math tasks span competition- and research-level problems, axiomatic proofs, broad mathematical reasoning, code-driven solving, and execution-based validation.The suite draws on OlympiadBench, amc12, aime, Putnam_Axiom, NuminaMath, OmniMath, MathCoder, and RealMath.
- A.1 Dataset Descriptions: Office tasks simulate desktop automation and enterprise document processing, including paragraph-level question answering, information retrieval, professional content generation, stylistic refinement, and formatting.OfficeQA and WritingBench provide the source tasks.
B.1 Pairwise Dependency Evaluation … C.3 Overall Skill Recurrence
The paper constructs a dependency-guided curriculum from bidirectional transfer judgments and validates whether ordered tasks reuse recurring core skills. Across domains, 69.5% of eligible tasks reuse a prior core skill, with 35.5% mean core-skill coverage.
- B.1 Pairwise Dependency Evaluation: 400 directional judgments per domain evaluate whether completing one task provides transferable skills for another, using YES, PARTIAL, or NO labels.Each of 200 sampled unordered task pairs is evaluated in both directions by GPT-5.4.
- B.1 Pairwise Dependency Evaluation: YES transfer judgments create directed dependency edges, while PARTIAL and NO judgments remain for analysis but do not become graph edges.An edge A →B indicates that Task A is expected to provide useful skills for Task B.
- B.2 Graph-based Ordering: Tasks are ordered from easier to harder curriculum levels, then greedily topologically sorted by largest outgoing degree to prioritize downstream skill usefulness.Higher-to-lower curriculum edges are removed; cycles are broken by selecting the remaining task with the largest outgoing degree.
- C Structural Validation of Skill Continuity: Structural validation examines whether curated sequences require recurring core skills within local history windows.The analysis tests skill continuity as a property of the ordered task sequences.
- C.1 Task-level Skill Annotation: Task annotations retain core reusable capabilities, yielding 318 unique core-skill names in Law, 335 in Finance, 345 in Healthcare, 262 in Mathematics, and 284 in Office.Skills are freeform annotations, so semantic matching accounts for differently named capabilities.
- C.1 Task-level Skill Annotation: Semantic skill matches combine normalized names with task-independent descriptions encoded by all-mpnet-base-v2, using cosine threshold τ = 0.85.Exact normalized-name matches are also included as semantic matches.
- C.2 Skill-continuity Metrics: The analysis measures task reuse rate and mean core-skill coverage over preceding task histories, with cross-domain results computed as macro-averages.Reuse rate counts tasks with at least one matched prior core skill; coverage measures the fraction of target requirements previously encountered.
- C.3 Overall Skill Recurrence: 69.5% of eligible tasks reuse at least one prior core skill, while mean core-skill coverage is 35.5% across the five domains.Domain-level reuse ranges from 63.6% to 77.8%, and coverage ranges from 23.2% in Healthcare to 46.2% in Office.
C.4 Comparison with Random Task Orders … E Example Task Format and Skill-Dependency Chain
The benchmark’s curated task order creates statistically reliable local opportunities for skill reuse, while the protocol operationalizes continual learning through execution, feedback, and skill updates. Sensitivity analysis and a Finance example show that reuse persists under stricter matching and can support increasingly complex procedures.
- C.4 Comparison with Random Task Orders: Curated ordering improved core-skill coverage over random permutations in all 15 domain–window comparisons, with ten remaining significant after Holm correction.The comparison permuted 100 tasks 10,000 times for history windows of 1, 5, and 10 tasks.
- C.4 Comparison with Random Task Orders: Finance had the largest gains: 5.6, 5.1, and 3.9 percentage points above the random-order mean for windows of 1, 5, and 10 tasks, respectively.Law improved significantly across all three windows, while Healthcare and Office showed significant improvements in some windows.
- C.4 Comparison with Random Task Orders: The ordered sequences therefore exhibit local skill continuity, but the analysis measures structural reuse opportunities rather than whether agents successfully exploit them.Nearby tasks are more likely to require overlapping or semantically related core skills than tasks under random ordering.
- C.5 Sensitivity to the Similarity Threshold: 63.8% of tasks reused at least one previous core skill even under the stricter semantic similarity threshold of 0.90.The macro-averaged conclusion remained stable across the tested thresholds, while exact normalized-name matching served as a conservative lower bound.
- D Full Prompts for the Three-Turn Interaction Protocol: Each sequential subtask used a three-turn protocol: task introduction with skill review, task execution, and reflection with evaluator feedback and repository updates.Agents could create new skills or revise existing ones, making updated skills available to subsequent subtasks.
- D Full Prompts for the Three-Turn Interaction Protocol: The maintenance instructions required agents to abstract feedback into reusable, modular skills rather than memorize task-specific solutions or hardcode fixes.This operationalized skill extraction and broad future applicability within the benchmark framework.
- E Example Task Format and Skill-Dependency Chain: The Finance dependency chain progresses from table-value retrieval in Tasks 1 and 7 to percentage growth, ratio interpretation, and programmatic WACC computation in Tasks 35, 50, and 81.Later tasks reuse earlier locating and matching operations as subroutines for more complex reasoning and validation.
F Additional Ablation: Retrieval-Augmented Trajectory Memory
Retrieval-augmented trajectory memory produces a pattern similar to in-context learning: it improves rubric-judged subtasks, while sequential execution remains stronger on exact-match rewards. This suggests contextual adaptation can help open-ended answers, whereas explicit skills better enforce strict output procedures.
- Method: RAG indexes and retrieves prior trajectory snippets for later subtasks instead of maintaining an explicit skill library.The baseline is evaluated in Table 7.
- Results: RAG improves rubric-judged subtasks, resembling the pattern observed in the ICL setting.This suggests part of the rubric gain may come from adapting to prior feedback and response patterns rather than reusable skill acquisition.
- Results: Seq. remains stronger on exact-match rewards, favoring explicit skill maintenance for strict output formats, answer normalization, and step-by-step verification.The comparison indicates a more disciplined workflow for tasks requiring precise outputs.
G Analysis on Generated Skills
Generated skills were evaluated for loadability and reusable content quality, with invalid formatting forcing the overall score to zero. GPT-4o generated many more but lower-quality skills, whereas GPT-5.3-Codex maintained a smaller, higher-quality pool, consistent with better consolidation of experience.
- Evaluation criteria: The evaluation judged whether each Skill abstracted a reusable method rather than memorizing a task, while including usage conditions, executable guidance, pitfalls, validation, and concise content.The evaluator was instructed to judge only the provided Skill content and penalize missing or truncated content.
- Evaluation criteria: Format/loadability was assessed through binary indicators for valid YAML front matter, a non-empty name, name matching, and a one-sentence description.These indicators determine whether the skill can be reliably loaded and reused.
- Scoring: Invalid formatting reduces the overall score to 0 even when the skill content is high quality, because the agent cannot reliably load or reuse it.The resulting evaluations are reported in Table 8.
- Model comparison: 384 skills and 5.68 average quality: GPT-4o produced a larger skill pool with substantially lower average quality than GPT-5.3-Codex’s 205 skills and 7.94 average.The comparison provides evidence for skill fragmentation and suggests that stronger models better consolidate repeated experience into compact, reusable abstractions.
H Examples of Generated Skills
The section illustrates generated skills from GPT-4o, GPT-5.3-Codex, and Claude 4.7 Opus, spanning data filtering, financial question answering, and medical advice. Stronger models tend to produce more structured workflows, explicit validation, and clearer awareness of pitfalls, helping explain larger and more reliable sequential gains in Finance and Healthcare.
- Examples of Generated Skills: Three generated skills cover date-range filtering, exact financial metric alignment, and structured open-ended medical advice with citations and conclusions.The examples come from GPT-4o, GPT-5.3-Codex, and Claude 4.7 Opus.
- Examples of Generated Skills: The date-range filter excludes the end date while enforcing YYYY-MM-DD inputs and trading-period requirements.Its implementation retains dates satisfying start_date <= date < end_date.
- Examples of Generated Skills: The financial QA skill prevents extraction mistakes by matching the requested metric, checking units and periods, and distinguishing line items, subtotals, and combined figures.Its workflow builds candidates from table and narrative sources before validating source selection and output order.
- Examples of Generated Skills: The medical-advice skill handles patient questions, including non-English prompts, and produces structured JSON containing findings, citations, conclusions, recommendations, and safety notes.The example emphasizes clinically accurate guidance and authoritative source citations.
- Examples of Generated Skills: Stronger models tend to generate more structured workflows, explicit validation steps, and clearer pitfall awareness, supporting larger and more reliable sequential gains in Finance and Healthcare.This relationship is presented as an explanation for model differences in sequential performance.