SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks

1 Introduction

• Continual skill learning uses a generate-store-reuse cycle to expand agent capabilities as prebuilt skills fall short on novel tasks.Skills encode task-specific instructions, workflows, and domain knowledge, and enable agents to handle specialized tasks beyond their base capabilities. They have become an open standard across major agent platforms, with communities sharing skills across domains (Ling et al., 2026); prior methods use varied settings, leaving their behavior and effectiveness difficult to assess systematically (Li et al., 2026; Xia et al., 2026; Ni et al., 2026).Source: §1 · p. 1
• SkillLearnBench includes 20 skill-dependent tasks across 15 sub-domains, with deterministic verification and multiple instances testing skill reusability.Tasks are grounded in a community-driven taxonomy and are reliably solvable with human-authored reference skills specifying the correct procedure.Source: §1 · p. 2
• The benchmark evaluates generated skill quality, execution-trajectory alignment, and final task accuracy and efficiency to diagnose where failures arise.Its layered design distinguishes problems in the skill specification from deviations during execution.Source: §1 · p. 2
• Generated skills improve no-skill agents, but the best method closes only about 45% of the gap to human-authored skills.Benefits are greater for tasks with clear, reusable workflows than for open-ended tasks, where skills can hurt; stronger LLMs do not consistently help. Iterative gains come primarily from external feedback, while self-feedback fails to improve with more iterations, and skill adoption and execution guidance remain common shortcomings.Source: §1 · p. 2
• The evaluation compares One-Shot, Self Feedback, Teacher Feedback, and Skill Creator methods spanning single-pass generation, self-revision, expert guidance, and structured generation.Together, these methods represent one-shot, self/teacher-feedback, and skill-creator approaches to learning from agent experiences.Source: §1 · p. 2

Reusable knowledge for LLM agents

• Prior work represents reusable procedural knowledge as code libraries or natural-language workflows, while Anthropic’s agent skills standardizes structured documents with activation conditions, steps, and domain knowledge.Wu & Zhang (2026) describe acquisition, representation, invocation, and refinement as skill lifecycle stages; SkillLearnBench targets acquisition by evaluating generation from task descriptions.Source: §2 · pp. 2–3

Continual learning through skill generation

• Agent continual learning accumulates external skills rather than updating model weights; methods span one-shot, refinement, experience-based, and pipeline strategies, but lack a standardized controlled-comparison benchmark.SkillsBench (Li et al., 2026) finds that one-shot skills provide no benefit on average, suggesting single-pass generation is insufficient.Source: §2 · p. 3

Benchmarks for agent skills

• Existing skill benchmarks mainly measure task completion with versus without skills, while SkillLearnBench evaluates generation through specification quality, trajectory alignment, and task outcome.Prior approaches use binary pass/fail outcomes and do not assess the generation process or what makes a skill specification effective beyond task success.Source: §2 · p. 3

3.1 Tasks and Skills

• Tasks are multi-instance procedural problems with deterministic pass/fail verification, designed to resist no-skill solving while remaining solvable with human-authored skills.The 20 tasks span six major categories and 15 sub-domains; 17 adapt SkillsBench tasks and three are newly designed. Multiple instances vary inputs while preserving task structure, so a skill generated once per task can be tested for reusability.Source: §3.1 · pp. 3–4, Table 1 · p. 4

3.2 Evaluation Framework

• The evaluation framework measures generated skills, the agent’s execution behavior using them, and final task outcomes, with metrics generally scaled to 0–100.These are Levels 1, 2, and 3, respectively.Source: §3.2 · p. 4

3.2.1 Level 1: Skill Quality

• Level 1 scores generated skills as textual artifacts on coverage, executability, and safety, without executing them.Executability averages completeness, determinism, consistency, and usability; safety averages six risk dimensions, with higher scores indicating lower risk.Source: §3.2.1 · p. 5

3.2.2 Level 2: Trajectory Analysis

• Level 2 assesses execution through trajectory alignment with an oracle and the fraction of generated skills actually invoked.Alignment averages key-point recall, correct execution order, and task-completion completeness; low usage may indicate irrelevant skills or agent preference not to rely on them.Source: §3.2.2 · p. 5

3.2.3 Level 3: Task Outcome

• Level 3 measures task accuracy using deterministic pass/fail verification and solving efficiency using total token consumption.The no-skill baseline and human-authored skills serve as accuracy references; lower token usage indicates more focused execution with less trial and error.Source: §3.2.3 · p. 5

3.3 Continual Learning Methods

• Four skill-generation baselines share the same task instruction and seed-instance input but use one-shot, self-feedback, teacher-feedback, or structured skill-creation workflows.One-Shot follows SkillsBench (Li et al., 2026) without feedback; Self Feedback follows SkillRL (Xia et al., 2026) through repeated self-review; Teacher Feedback uses expert guidance after failures (Huang et al., 2025); Skill Creator follows Anthropic’s pipeline.Source: §3.3 · p. 6

4.1 Experimental Setup

• The comparison spans four methods across six LLMs, with No Skill and Human-authored baselines, and evaluates every skill using a fixed Claude Sonnet 4.6 solver.Self Feedback uses K = 2 and Teacher Feedback K = 3; GPT-5-mini judges Level 1 and Level 2 metrics requiring an LLM judge.Source: §4.1 · p. 6

4.2 Main Results

• Across LLMs, all four methods beat No Skill and mostly reduce token cost versus Human-authored, but remain less accurate; Self Feedback leads on average, with One-Shot close behind.Coverage and Alignment are low across methods, while Teacher Feedback has the lowest Usage, suggesting success depends on both skill content and solver adoption.Source: Table 2 · p. 7, §4.2 · p. 7
• Method rankings shift across LLMs and model families, and stronger LLMs do not reliably produce better skills.The continual-learning LLM choice largely determines which method works best.Source: §4.2 · p. 7

5.1 Learning Effect Across Categories

• Continual learning gains were largest for tasks with clear workflows, while open-ended or instance-specific tasks sometimes regressed, and no method led across all categories.Token cost varied mainly by category, with similar costs among the four methods within each category; Productivity Tools, Data & Analytics, and Content used less across methods.Source: §5.1 · p. 8

5.2 Skill Reusability

• Most generated skills passed some instances but failed on others, and held-out accuracy comparable to or slightly above seed-instance accuracy ruled out seed overfitting as the bottleneck.The remaining gap was failure to capture core task logic needed for consistent success across instances.Source: §5.2 · p. 8

5.3 Learning Effect on Agent Behavior

• Relative to One-Shot, feedback mechanisms reshaped agent behavior differently, and behavioral shifts did not always align with accuracy improvement.Self Feedback produced shorter trajectories with fewer steps and tool calls alongside slightly better accuracy; Teacher Feedback raised coverage but lowered adoption and increased execution burden, while Skill Creator’s frequently invoked structured skills kept execution lean but lagged in accuracy.Source: §5.3 · p. 9, Figure 3 · p. 9

5.4 Skill Evolution

• Across four rounds, Teacher Feedback steadily increased accuracy while maintaining alignment, whereas Self Feedback’s accuracy fell after an initial rise as alignment declined.Teacher Feedback’s first round raised coverage and later gains compounded; without new external information, Self Feedback reshuffled skill content without fixing underlying gaps, leading to drift from correct execution paths.Source: §5.4 · p. 9, Figure 4 · p. 9

6 Conclusion

• All four evaluated continual-learning methods improve over the no-skill baseline, but remain well below human-authored performance.The benchmark covers 20 verified tasks across 15 sub-domains and evaluates skill quality, execution trajectory, and task outcome.Source: §6 · p. 10
• Skill learning yields the largest gains on tasks with clear, reusable workflows but can hurt open-ended tasks by overconstraining agents.The generation LLM shapes outcomes, and larger models do not consistently produce better skills.Source: §6 · p. 10
• External feedback supports genuine improvement across refinement rounds, whereas self-feedback without new signals leads to drift rather than progress.The conclusion identifies grounding skills in core task logic and ensuring agents reliably adopt and follow them as priorities for advancing skill generation.Source: §6 · p. 10

Contents of Appendix

• The appendix examines One-Shot seed-instance counts and the distribution of generated-skill patterns.The analyses cover Productivity Tools with Claude Sonnet 4.6 and patterns averaged across generation methods and LLMs.Source: Table 4 · p. 14, Table 5 · p. 14

A The Effect of Instance Count on Skill Generation

• For One-Shot on Productivity Tools with Claude Sonnet 4.6, increasing seed instances from 1 to 3 to 5 does not improve performance and accuracy drops sharply from 1 to 3.More instances yield fewer skills with lower coverage; the authors suggest broader but shallower skills dilute actionable guidance, making depth on one instance more valuable than breadth.Source: A The Effect of Instance Count on… · p. 14, Table 4 · p. 14

B Pattern Analysis of Generated Skills

• Nearly all generated skills are Pattern A: One-Shot and Teacher Feedback are 100% Pattern A, Self Feedback has 0.3% Pattern B, and Skill Creator has 1.1% Pattern B.No method produces Pattern C; human-authored skills are 64.3% Pattern A, 33.3% Pattern B, and 2.4% Pattern C, indicating a gap in structural diversity and executable artifacts.Source: B Pattern Analysis of Generated Skills · p. 14, Table 5 · p. 14

C The Effect of Generation LLM Scale on Skill Quality

• Scaling does not monotonically improve accuracy; Figure 5 shows scale reshapes the accuracy–cost trade-off rather than uniformly improving skill quality.Source: C The Effect of Generation LLM Scale… · pp. 14, 16, Figure 5 · p. 15
• Opus’s precise but hardcoded skill earns higher Level 1 coverage yet fails on varying instances, while Sonnet’s dynamic skill remains flexible.Source: Figure 6 · p. 15, C The Effect of Generation LLM Scale… · p. 16
• Sonnet is often competitive or better in Claude, while Flash and Pro both beat Flash Lite in Gemini without a clear largest-model advantage.Source: C The Effect of Generation LLM Scale… · p. 15
• Method rankings shift across LLM families: Teacher Feedback is stable in Claude but less reliable in Gemini, where Self Feedback and Skill Creator can surpass it.Source: C The Effect of Generation LLM Scale… · p. 15
• Larger models generally reduce solving tokens across both families, except Gemini One-Shot, whose cost rises with scale.Source: C The Effect of Generation LLM Scale… · p. 15

D Token Efficiency and Cost Breakdown in Skill-Based Solving

• Across all LLMs, input tokens exceed 97% of cost; all four methods reduce total tokens versus No Skill and Human-authored, though Human-authored costs slightly more despite higher accuracy.Source: D Token Efficiency and Cost Breakdown… · p. 16
• Self Feedback has the lowest total cost and highest accuracy; Skill Creator costs similarly with lower accuracy, while Teacher Feedback nears No Skill’s cost.Source: D Token Efficiency and Cost Breakdown… · p. 16
• Output tokens remain small and stable; efficiency gains come from reduced input context, while Teacher Feedback’s longer, more detailed skills inflate it.Source: D Token Efficiency and Cost Breakdown… · p. 16

E Robustness of Single-Trial Evaluation

• Across three trials, One-Shot and Self Feedback lead in mean accuracy, preserving the relative method ordering from single-trial results.The robustness analysis uses one representative seed instance per task and three solving runs per method and skill-generation LLM.Source: E Robustness of Single-Trial Evaluation · p. 16, Table 7 · p. 17
• Trajectory alignment remains stable across trials for all methods, with standard deviation around 10, indicating consistent execution paths.The passage attributes this stability to skill quality reliably shaping the agent’s execution path.Source: E Robustness of Single-Trial Evaluation · p. 16, Table 7 · p. 17
• Generated-method accuracy varies more across trials, with standard deviation of 13–17%, as small execution differences can determine verification success.Agents can follow consistent trajectories while differing slightly in execution and final output.Source: E Robustness of Single-Trial Evaluation · p. 16, Table 7 · p. 17
• The three-trial analysis confirms that single-trial evaluation provides reliable method comparisons.Table 7 reports mean and standard deviation across three trials.Source: E Robustness of Single-Trial Evaluation · p. 16, Table 7 · p. 17

F Per-Task Accuracy and Skill Usage Analysis

• No single generation method dominates across tasks, and the best method varies by task.Per-task accuracy is averaged across all LLMs.Source: F Per-Task Accuracy and Skill Usage… · p. 17
• Accuracy with human-authored skills spans from near zero to perfect, reflecting tasks that vary widely in difficulty.Some tasks are inherently challenging, while high-quality skills can reliably solve others.Source: F Per-Task Accuracy and Skill Usage… · p. 17

• Teacher Feedback and Skill Creator achieve the best average completeness, while Teacher Feedback also leads in determinism, at the cost of some consistency and usability.Source: G Executability Results Across All… · p. 17
• One-Shot performs best in consistency and maintains relatively high usability, but has lower completeness and stability; Self-Feedback is balanced without leading in any dimension.Source: G Executability Results Across All… · p. 17
• Stronger LLMs benefit more from complex methods, while weaker models show limited gains; simpler methods favor consistency and usability, and complex methods favor completeness and determinism.The passage characterizes model capability as setting the upper bound.Source: G Executability Results Across All… · p. 17
• Completeness and determinism scores are relatively low across skills and models, hypothesized to result from skills emphasizing high-level concepts over concrete, step-by-step actions.Source: G Executability Results Across All… · p. 17

H Safety Results Across All Dimensions

• Bias and Illegal/Offensive Content are nearly saturated across methods, while One-Shot performs best on average in Privacy, Prompt Injection, and System Integrity.Self-Feedback remains competitive but does not clearly outperform other methods.Source: H Safety Results Across All Dimensions · p. 18
• Skill Creator excels in Bias, Illegal/Offensive Content, and Untrusted Communication but trades off slightly in Privacy and Integrity; Teacher Feedback underperforms, especially in Prompt Injection and Integrity.The passage suggests added complexity may introduce vulnerabilities.Source: H Safety Results Across All Dimensions · p. 19
• Overall, simpler methods tend to be more robust, while complex methods gain in specific areas at the cost of consistency.Source: H Safety Results Across All Dimensions · p. 19

I Trajectory Alignment Results Across All Dimensions

• Across all tasks, skill-generated trajectories consistently outperform those without skills, but the optimal method varies across LLMs.Table 12 presents the full trajectory-alignment results, including three component dimensions and an aggregated score.Source: I Trajectory Alignment Results Across… · p. 19, Table 12 · p. 21
• Trajectory keypoint recall has the largest gap across dimensions, while completeness differences are relatively small, suggesting agents usually reach a valid end state.Source: I Trajectory Alignment Results Across… · pp. 19–20
• Claude Opus 4.6 under the Teacher Feedback learning paradigm achieves the strongest trajectory-alignment performance.Source: I Trajectory Alignment Results Across… · p. 20

J Case Studies

• Teacher Feedback achieves the highest video-object-counting accuracy despite the lowest coverage, while every higher-coverage method shares the same low accuracy.Its iterative refinement produces working parameters, whereas One-Shot leaves parameters unspecified and the agent fails to discover the matching threshold.Source: J Case Studies · p. 22
• On dbscan-parameter-tuning, most generated methods outperform No Skill despite covering less than half the oracle key points, while human-authored skills achieve perfect accuracy.One-Shot supplies a task-specific pipeline with dataset-specific columns and evaluation logic, complementing the human-authored skill’s generic reusable components.Source: J Case Studies · pp. 23–24, Table 14 · p. 23, Figure 8 · p. 23
• On temperature-simulation, Skill Creator has the highest generated-method coverage and exceeds human-authored skills in alignment and usage, yet remains far less accurate.The generated skills describe the calibration workflow, but without code-level RMSE computation and parameter search, the agent must improvise under a 100-turn and 1800-second budget; human-authored code support succeeds.Source: Table 15 · p. 24, J Case Studies · pp. 24–25, Figure 9 · p. 24

K Details of Data Collection

• The benchmark retains tasks where agents succeed with skills but fail without them, balances remaining candidates across categories and sub-domains, and excludes structurally simple tasks still solvable without specialized skills.For robustness, it checks similar instances and rephrased or compressed instructions. The benchmark contains 20 tasks: 17 adapted from SkillsBench and three newly created.Source: §K.1 · p. 25, §K.2.1 · p. 25, §K.2.2 · pp. 25–26

K.3 Human-authored Collection and Construction

• Human-authored skills are curated for discoverability and effectiveness, then instance augmentation tests generalization through semantic rephrasing, instruction changes, and input-data variations.Refinements improve task alignment and invocation; input variations include formats, content, and distracting data.Source: §K.3 · p. 26, §K.4 · p. 26, §K.4.1 · p. 26

L.1.1 Coverage

• Coverage is the fraction of human-authored key points judged mentioned in generated skills; executability assesses completeness, determinism, consistency, and reusability.Key points are task-essential units extracted from human-authored skills; vague or partial coverage counts as missing.Source: §L.1.1 · pp. 27–28, L Implementation Details of Evaluation… · p. 27, §L.1.2 · p. 29

L.1.3 Safety

• The safety evaluator scores generated skills from 1–5 across six risk surfaces, with higher scores indicating safer behavior.The dimensions cover privacy, prompt injection, harmful content, discrimination, system integrity, and untrusted communication.Source: §L.1.3 · p. 29

L.2 Level 2: Trajectory Analysis

• Skill use is the fraction of generated skills invoked during execution, while trajectory alignment averages keypoint recall, execution order, and completeness.Invocation is detected through skill tool calls; order and completeness are scored on a normalized 0–100 scale, as is keypoint recall, and the three components are averaged.Source: §L.2 · pp. 29–30

L.3 Judge Reliability

• Executability means showed test–retest agreement of ρ=0.76, while safety means showed ρ=0.61 and small drift (MAE 0.10).The identical scoring pipeline was run twice on three tasks; paired runs were compared using per-run means of four executability and six safety rubrics.Source: Table 16 · p. 30, §L.3 · p. 30

L.4 Tool Calls and Trajectory Steps

• Trajectory steps count unique assistant turns after deduplication, while tool calls count all invocations across the trajectory.Longer trajectories may reflect complex problem-solving, iterative refinement, or planning inefficiency.Source: §L.4 · p. 31

M Implementation Details of Continual Learning

• The four baselines frame continual learning as skill-level learning without parameter updates, differing in single-pass acquisition, self- or teacher-guided revision, and structured skill authoring.One-Shot, Self Feedback, and Skill-Creator use prompt-injected instructions; Teacher Feedback uses a runner-coordinated multi-round teacher–student loop.Source: M Implementation Details of Continual… · p. 31

M.1 One-Shot

• One-Shot distills task knowledge into reusable skill documents and solves the task using them, without revising the skills after execution (Li et al., 2026; Mi et al., 2026).It provides a lower bound on skill-level knowledge acquisition without reflection, critique, or iterative consolidation.Source: §M.1 · p. 31

M.2 Self Feedback

• Self Feedback runs an initial generate-and-solve round, then uses the agent’s prior execution experience to revise skills and solve again (Xia et al., 2026; Yang et al., 2026).The default configuration uses two rounds and preserves a complete skill-set snapshot for each round.Source: §M.2 · pp. 31–32

M.3 Teacher Feedback

• Teacher Feedback uses an external teacher’s concise modification suggestions to guide student skill revisions across three rounds, rather than relying only on the student’s execution trace (Huang et al., 2025).The teacher can see the task, current student skill, failure context, and human-authored skill but cannot provide the full solution.Source: §M.3 · pp. 32–33

M.4 Skill-Creator

• Skill-Creator keeps One-Shot’s acquire-then-reuse workflow but adds structured guidance for packaging knowledge into reusable SKILL.md documents (Anthropic, 2025c;b; Liang et al., 2026; Anthropic, 2025b).The method encourages YAML frontmatter, precise descriptions, and trigger-oriented documentation; it uses one round without teacher feedback or iterative reflection.Source: §M.4 · p. 33

N Implementation Details of Solving Agent

• Each task is evaluated by a capable coding agent in a sandbox, with its output scored by a deterministic verifier.The appendix describes the agent setup and verifier design in turn.Source: N Implementation Details of Solving… · p. 33

N.1 Solving Agent

• The solving agent runs in a Docker sandbox with skills injected as environment files, while its prompt remains the original task instruction to keep the solver interface fixed across methods.Trials are limited to 100 turns and 1,800 seconds; after a timeout, the verifier still runs on the output produced so far.Source: §N.1 · pp. 34–35

N.2 Verifier Details

• The verifier runs each task’s read-only test.sh unconditionally after agent termination and marks success only when all test cases pass.Tasks average 10.6 test cases (median 7; range 1–47), covering file existence, numerical comparisons, structured outputs, and functional correctness.Source: §N.2 · p. 35

O Complete Instance List

• The instance list covers base tasks and variants created through instruction-level changes, input-data variation, and semantic rephrasing.Examples include changing travel preferences, modifying financial-analysis targets, and contextualizing tasks as exam-style problems.Source: O Complete Instance List · pp. 36–40, Table 17 · p. 41

P Task Examples

• The task-example section presents a complete task specification and representative instances produced by the benchmark’s augmentation strategies.It directs readers to a financial-analysis example and examples of semantic rephrasing, instruction modification, and input-data variation.Source: P Task Examples · p. 41

P.1 Example Tasks

• The financial-analysis example includes a task instruction, human-authored skill headers, and a test suite for analyzing hedge-fund activity across quarters.Its questions cover Renaissance Technologies’ AUM and holdings, Berkshire Hathaway’s increased investments, and fund managers’ Palantir holdings.Source: §P.1 · pp. 41–42

P.2 Augmentation Instances

• The augmentation examples show semantic rephrasing through contextual transformation, instruction changes to task parameters or constraints, and input-data variations.The examples respectively recast a task as an exam-style problem, alter task-specific parameters, and modify instance fields or values.Source: §P.2 · p. 43, Figure 18 · p. 44, Figure 19 · p. 45