Source-linked AI summary
SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks
Shanshan Zhong, Yi Lu, Jingjie Ning, Yibing Wan, Lihan Feng, Yuyi Ao, Leonardo F. R. Ribeiro, Markus Dreyer, Sean Ammirati, Chenyan Xiong
TL;DR
Automatic learning of reusable skills for LLM agents remains insufficiently understood, especially as agents encounter novel tasks. The paper introduces SkillLearnBench, a benchmark with verified skill-dependent tasks and three evaluation levels, and compares continual skill-generation methods. All methods improve over no-skill baselines, but gains are inconsistent across tasks and models, with external feedback supporting improvement more reliably than self-feedback.
Problem
How to learn reusable skills automatically and effectively for agents facing novel tasks remains unclear, while existing evaluation often focuses mainly on task completion.
Method
SkillLearnBench evaluates continual skill learning on 20 verified tasks across 15 sub-domains at the levels of skill quality, execution trajectory, and task outcome.
Results
All continual learning methods improve over no-skill baselines, but none consistently leads across tasks and LLMs, and stronger LLMs do not reliably produce better skills.
Takeaways & Limitations
Continual skill learning is most useful for clear, reusable workflows, while external feedback enables iterative gains and self-feedback alone can induce recursive drift.
Takeaways & Limitations
Each benchmark task is required to genuinely require skills and be solvable with them, constraining the evaluation to verified skill-dependent settings.
Abstract
from arXiv · showhide
Skills have become the de facto way to enable LLM agents to perform complex real-world tasks with customized instructions, workflows, and tools, but how to learn them automatically and effectively remains unclear. We introduce SkillLearnBench, the first benchmark for evaluating continual skill learning methods, comprising 20 verified, skill-dependent tasks across 15 sub-domains derived from a real-world skill taxonomy , evaluated at three levels: skill quality, execution trajectory, and task outcome. Using this benchmark, we evaluate recent continual learning techniques, those leveraging one-shot, self/teacher feedback, and skill creator to generate skills from agent experiences. We find that all continual learning methods improve over the no-skill baseline, yet consistent gains remain elusive: no method leads across all tasks and LLMs, and scaling to stronger LLMs does not reliably help. Continual learning improves tasks with clear, reusable workflows but struggles on open-ended tasks, and using stronger LLM backbones does not consistently produce better skills. Our analysis also revealed that multiple iterations in continual learning facilitate genuine improvement via external feedback, whereas self-feedback alone induces recursive drift. Our data and code are open-source at https://github.com/cxcscmu/SkillLearnBench to enable further studies of automatic skill generation and continual learning techniques.
1 Introduction
SkillLearnBench introduces a benchmark for continual skill learning and evaluates how generated skills affect skill quality, execution, and task outcomes. Its comparison finds improvements over no-skill baselines, but performance depends on task structure, feedback, and the generation model.
- Benchmark: SkillLearnBench is the first benchmark for evaluating continual learning methods that generate skills from agent experiences.It contains 20 verified, skill-dependent tasks across 15 sub-domains, with multiple instances testing reusability.
- Evaluation framework: The benchmark evaluates generated skill specifications, execution trajectories, and final task outcomes to diagnose where methods succeed or fail.The three levels assess skill quality, alignment with expected steps, and task accuracy and solving efficiency.
- Methods: Four continual learning methods—One-Shot, Self Feedback, Teacher Feedback, and Skill Creator—are compared using diverse skill-generation strategies.The methods respectively use single-pass generation, self-revision, iterative expert guidance, and structured pipeline generation.
- Findings: All evaluated methods improve over no-skill baselines, but the best method covers only about 45% of the gap to human-authored performance.The methods remain well below human-authored skills, and stronger LLMs do not always close the gap.
- Findings: Continual skill learning helps most on tasks with clear, reusable workflows and can hurt performance on open-ended tasks.The paper also reports that external feedback supports iterative gains, whereas self-feedback methods fail to improve with additional iterations.
- Findings: Generated skills often fail because they omit core content or are not adopted correctly during agent execution.The benchmark’s layered evaluation identifies both low skill coverage or alignment and insufficient skill usage as sources of limited gains.
2 Related Work
Prior work develops reusable knowledge representations, continual skill-generation methods, and completion-based skill benchmarks. SkillLearnBench extends these lines by evaluating the skill, its invocation and execution, and the resulting task outcome.
- Reusable knowledge for LLM agents: Reusable agent knowledge has been represented as executable code, reusable functions, natural-language insights, and workflows induced from prior trajectories.These approaches demonstrate the value of reusable procedural knowledge while using different representations.
- Reusable knowledge for LLM agents: A recent survey frames skills as procedural memory with acquisition, representation, invocation, and refinement stages.This paper focuses on acquisition, specifically methods that generate skills from task descriptions.
- Continual learning through skill generation: Continual skill generation methods include one-shot generation, refinement through execution feedback, and experience-based conversion of interaction traces into reusable skills.Examples include SkillsBench, SkillRL, EvoSkill, and ProcMEM, which differ in how they create or improve skills.
- Benchmarks for agent skills: Existing skill benchmarks primarily measure whether adding a skill changes task completion, often using with-versus-without comparisons.Reported challenges include unreliable skill invocation and trade-offs in skill granularity.
- Benchmarks for agent skills: SkillLearnBench addresses this evaluation gap by examining skill quality, execution trajectories, and task outcomes rather than task completion alone.Its layered evaluation distinguishes failures in the generated skill from deviations during execution.
3 SkillLearnBench
SkillLearnBench benchmarks continual skill learning with verified, skill-dependent tasks and evaluates generated skills across quality, execution behavior, and task outcomes. Its design tests whether skills are reusable across instances and compares multiple skill-generation strategies.
- Benchmark design: SkillLearnBench combines a curated task collection, human-authored skills, multiple instances, three evaluation levels, and four continual-learning baselines.The benchmark evaluates skill quality, execution behavior, and final task outcomes.
- Tasks and skills: Each task is a multi-step, verifiable procedural problem with multiple solvable instances in a controlled environment and a specified goal state.Tasks use deterministic verification rather than subjective grading.
- Skill dependency verification: Tasks are verified as skill-dependent by requiring low no-skill pass rates and successful completion with human-authored skills.The no-skill pass rate must remain below α, with R = 10 and α = 0.5; each instance must have at least one successful human-authored-skill completion.
- Tasks and skills: The benchmark spans 6 major categories and 15 sub-domains derived from community skill-usage patterns.The task collection contains 20 tasks, including adapted and newly created tasks.
- Tasks and skills: Multiple instances vary parameters while preserving core task structure, allowing one generated skill to be tested for reusability across instances.Skills are generated once per task and then evaluated on all instances.
- Continual learning methods: The four methods are One-Shot, Self Feedback, Teacher Feedback, and Skill Creator, all mapping the task instruction and one seed instance to a generated skill set.One-Shot generates skills once, while Self Feedback iteratively revises skills using execution without external supervision.
4 Experiments
Experiments compare four continual-learning methods across six LLMs using a fixed solving agent and metrics spanning execution and task outcomes. All methods improve over no-skill baselines, but performance varies by method, model, and task category.
- Experimental setup: Four methods are evaluated across six LLMs from the Claude and Gemini families, alongside No Skill and Human-authored reference conditions.The methods are One-Shot, Self Feedback, Teacher Feedback, and Skill Creator.
- Experimental setup: A fixed Claude Sonnet 4.6 solving agent at temperature 0 isolates differences attributable mainly to generated skills.The solver runs for up to 100 turns in a containerized sandbox.
- Overall performance: All four continual-learning methods outperform No Skill, while Self Feedback performs best on average and uses the fewest tokens.One-Shot is close behind, whereas Teacher Feedback and Skill Creator lag; generated methods remain far below Human-authored accuracy.
- Overall performance: Method rankings shift across LLMs and families, and stronger LLMs do not reliably produce better skills.The choice of continual-learning LLM largely determines which method works best.
- Skill reuse: Most generated skills are partially effective across instances, while held-out accuracy is comparable to or slightly better than seed-instance accuracy.This pattern rules out seed-instance overfitting as the explanation for the observed inconsistency.
5 Analysis
Skill learning helps most on tasks with reusable workflows, but generated skills often generalize only partially and reshape agent behavior in method-specific ways. External teacher feedback improves skills across rounds, whereas repeated self-revision can cause drift.
- Task Categories: Continual learning yields larger gains on tasks with clear workflows and smaller or negative gains on open-ended, instance-specific tasks.Reusable plans reduce trial and error, while rigid skills can overconstrain agents.
- Skill Reusability: Generated skills pass some instances but fail others because they miss core task logic rather than simply overfitting seed instances.Held-out accuracy is comparable, ruling out seed-instance overfitting as the main explanation.
- Agent Behavior: Self Feedback produces shorter trajectories with fewer steps and tool calls, alongside slightly better accuracy.Its execution becomes more focused through execution-based revision.
- Agent Behavior: Teacher Feedback increases coverage but lowers adoption and makes execution heavier as agents navigate more instructions.The method expands the skill set through iterative teacher guidance.
- Skill Evolution: Repeated Self Feedback leaves coverage flat, decreases alignment, and causes accuracy to fall after an initial rise.Teacher Feedback instead uses external information to support genuine improvement across refinement rounds.
6 Conclusion
The conclusion presents SkillLearnBench as a layered benchmark for continual skill learning and finds that generated skills improve over no-skill solving but remain far below human-authored performance. Benefits depend on task structure, model compatibility, and externally grounded refinement.
- Benchmark: SkillLearnBench evaluates continual skill learning on 20 verified tasks across 15 sub-domains using skill quality, execution trajectory, and task outcome.The benchmark tests whether generated skills are reusable across multiple task instances.
- Findings: All four continual learning methods improve over the no-skill baseline, but none consistently dominates across tasks and models.The best method varies by task, and generated skills remain below human-authored performance.
- Ablations: Increasing One-Shot seed instances from 1 to 3 to 5 does not improve performance and sharply lowers accuracy from 1 to 3 instances.The generator produces fewer skills with lower coverage as more instances are supplied.
- Skill Forms: Nearly all generated skills are Pattern A instructions-only artifacts, while no method produces Pattern C skills with MCP or subagents.Skill Creator has the highest Pattern B share at 1.1%.
C The Effect of Generation LLM Scale on Skill Quality
Generation-model scale changes accuracy and cost unevenly rather than producing uniformly better skills. Method effectiveness depends strongly on compatibility with the underlying LLM, while stronger models can create more prescriptive skills that fail to generalize.
- Accuracy Scaling: Stronger models do not reliably improve skill accuracy, and mid-tier models are often competitive with larger models.In Claude, Sonnet is frequently competitive or better than Haiku and Opus; in Gemini, Pro has no clear advantage over Flash.
- Method–Model Interaction: Method rankings shift across Claude and Gemini, showing that continual-learning effectiveness depends strongly on the generation LLM.Teacher Feedback is stable in Claude but less reliable in Gemini, while Skill Creator improves substantially with stronger Gemini models.
- Cost Scaling: Larger models generally reduce solving tokens even when accuracy gains are limited, except for One-Shot in Gemini.Scaling therefore reshapes the accuracy–cost trade-off rather than uniformly improving performance.
- Skill Quality: Stronger models tend to produce more precise but prescriptive skills that hardcode parameters, library choices, or field names.The earthquake-plate case contrasts Opus’s brittle hardcoding with Sonnet’s flexible dynamic detection and general-purpose projection.
- Token Efficiency: Input tokens exceed 97% of solving cost, while generated skills reduce total tokens by shortening context accumulation from turns, tool outputs, and history.Self Feedback has the lowest cost and highest accuracy; Teacher Feedback is most expensive.
E Robustness of Single-Trial Evaluation
Three-trial evaluation preserves the main method ordering and shows stable trajectory alignment, supporting the robustness of the single-trial setting. The benchmark still spans tasks with widely varying difficulty and no universally best generation method.
- Multi-Trial Validation: Three repeated trials preserve the relative method ordering, with One-Shot and Self Feedback leading in mean accuracy.The reported means and standard deviations validate the single-trial comparison.
- Multi-Trial Validation: Trajectory alignment remains stable across trials for all methods, with standard deviation around 10.This supports the consistency of skill effects on execution behavior.
- Task Difficulty: Human-authored-skill accuracy ranges from near zero to perfect across tasks, confirming widely varying benchmark difficulty.The suite includes both inherently challenging tasks and tasks that high-quality skills can reliably solve.
- Method Comparison: No single generation method dominates across all tasks, so the best method varies by task.Per-task results are averaged across all LLMs.
G Executability Results Across All Dimensions
Executability results reveal trade-offs among completeness, determinism, consistency, and usability. More complex methods improve structural dimensions, while simpler methods favor consistency and usability.
- Teacher Feedback and Skill Creator achieve the best completeness, while Teacher Feedback also leads in determinism.These methods sacrifice some consistency and usability.
- One-Shot performs best in consistency and maintains relatively high usability, but has lower completeness and stability.
- Stronger LLMs benefit more from complex methods, while weaker models show limited gains.Model capability sets the upper bound on executability performance.
- Completeness and determinism scores remain relatively low across all skills and models.The authors hypothesize that generated skills rely on high-level concepts rather than concrete, step-by-step actions.
H Safety Results Across All Dimensions
Safety performance is generally high, but meaningful differences emerge in privacy, prompt injection, system integrity, and untrusted communication. Simpler methods are generally more robust, whereas complex methods gain in selected safety areas at the cost of consistency.
- Bias and Illegal/Offensive content scores are nearly saturated across methods, offering limited differentiation.
- One-Shot performs best in Privacy, Prompt Injection, and System Integrity, while Self-Feedback remains competitive without clearly outperforming other methods.
- Skill Creator excels in Bias, Illegal/Offensive content, and Untrusted Communication but shows slight trade-offs in Privacy and Integrity.
- Teacher Feedback consistently underperforms in several dimensions, particularly Prompt Injection and Integrity.
- Overall, simpler methods tend to be more robust, while complex methods achieve gains in specific areas at the cost of consistency.
I Trajectory Alignment Results Across All Dimensions
Skills consistently improve trajectory alignment over no-skill execution, but the best generation method varies across LLMs. The largest remaining gap concerns trajectory keypoint recall rather than reaching a valid end state.
- Skill-conditioned trajectories consistently outperform trajectories without skills across all tasks.
- No single generation approach consistently achieves the best trajectory alignment across LLMs.
- Trajectory keypoint recall shows the largest gap among the three alignment dimensions, while completeness differences are relatively small.Agents can usually reach a valid end state.
- Claude Opus 4.6 with Teacher Feedback achieves the strongest trajectory-alignment performance.
J Case Studies
The case studies show that skill utility depends on actionable task-specific knowledge and executable support, not simply coverage or workflow description. The benchmark broadens evaluation through varied inputs and task-essential key points.
- Case 1: Lower Coverage but Higher Accuracy: Teacher Feedback achieves the highest video-object-counting accuracy despite the lowest coverage.Other methods cover more oracle key points but achieve the same low accuracy.
- Case 1: Lower Coverage but Higher Accuracy: One-Shot leaves video-object-counting parameters unspecified, whereas Teacher Feedback hardcodes working parameters after iterative refinement.The operational details resolve threshold decisions that generic algorithmic descriptions leave to the agent.
- Case 2: Task-Specific Pipelines Compensate for Low Coverage: Most generated methods substantially outperform No Skill on dbscan-parameter-tuning despite covering less than half of the oracle key points.Human-authored skills still achieve perfect accuracy.
- Case 2: Task-Specific Pipelines Compensate for Low Coverage: One-Shot embeds a full DBSCAN pipeline with dataset-specific column names and evaluation logic, unlike the human-authored generic building blocks.
- Case 3: Workflow Knowledge Without Executable Support: Skill Creator achieves the highest generated-method coverage and higher alignment and usage than human-authored skills, yet far lower temperature-simulation accuracy.
- Case 3: Workflow Knowledge Without Executable Support: Temperature-simulation skills describe calibration workflows faithfully, but lack executable support for systematic search across a multi-dimensional parameter space.Each trial requires running a full simulation and computing RMSE.
- Benchmark Construction: The benchmark uses 20 tasks and varied input formats, content, and noisy information to test skill behavior across instances.
- Benchmark Construction: Human-authored skill key points are atomic, nonoverlapping, and task-essential units covering procedure steps, rules, or usage patterns.
L.2 Level 2: Trajectory Analysis
Trajectory analysis measures whether generated skills guide agents toward oracle-like execution, while complementary metrics track skill invocation and trajectory length. The benchmark also documents four skill-generation baselines and their differing refinement workflows.
- Trajectory Metrics: Skill usage rate measures the average fraction of generated skills invoked during task execution.A skill counts as invoked when the agent explicitly reads, references, or applies it through a skill tool call.
- Trajectory Metrics: Trajectory alignment compares generated-skill executions with oracle trajectories using keypoint coverage, execution order, and completeness.The three dimensions diagnose whether agents cover expected actions, preserve sequencing, and finish with a task result.
- Trajectory Metrics: Trajectory alignment combines its three components by averaging normalized scores on a 0–100 scale.Execution order and completeness use 1–5 ratings normalized to 0–100, while trajectory keypoint is scaled from 0–1 by multiplying by 100.
- Trajectory Metrics: Trajectory steps count unique assistant turns, while tool calls count all tool invocations across those turns.Longer trajectories may reflect complex problem-solving, iterative refinement, or planning inefficiency.
- Compared Methods: The benchmark compares One-Shot, Self Feedback, Teacher Feedback, and Skill Creator as non-parametric continual-learning baselines.These methods differ in how they write and refine reusable external skills rather than updating model parameters.
- Compared Methods: One-Shot creates reusable skills once, Self Feedback revises them after execution, Teacher Feedback uses external guidance, and Skill Creator supplies structured authoring guidance.The default configurations use one, two, and three rounds for One-Shot, Self Feedback, and Teacher Feedback respectively; Skill Creator is single-round.
N Implementation Details of Solving Agent
The solving agent runs each task in a reproducible Docker sandbox and is evaluated by a deterministic verifier. Benchmark instances vary instructions, data, prerequisites, and task settings across concrete real-world examples.
- Agent and Verifier: Each task instance runs a capable coding agent in a sandbox and scores its output with a deterministic verifier.The setup separates agent execution from task-specific verification.
- Agent and Verifier: Docker images contain each task’s assets, libraries, and dependencies before the CLI agent is installed and executed at runtime.A long-lived container is launched for the trial, after which the verifier runs on the produced output.
- Agent and Verifier: Agents are limited to 100 turns and 1,800 seconds per trial, with verification performed even when a timeout terminates execution.The verifier scores whatever output exists when the agent stops.
- Agent and Verifier: A read-only test.sh script runs unconditionally after termination and writes a binary reward to /logs/verifier/reward.txt.The task is marked successful only when the task-specific test suite accepts the agent’s outputs.
- Instance Construction: Benchmark instances are created through base-instance, instruction-level, input-data, and semantic-rephrasing modifications.Examples alter prerequisites, filtering criteria, data values, objectives, wording, or contextual framing.
- Task Example: The financial-analysis example specifies quarterly hedge-fund analysis and questions about Renaissance Technologies’ AUM and stock holdings.The task uses Q2 and Q3 2025 datasets stored in separate directories and requires intermediate searches for fund details.
- Task Example: The Python-to-Scala task requires translating Tokenizer.py into readable, maintainable Scala 2.13 while preserving classes, functions, APIs, and behavior.The agent must consult TokenizerSpec.scala to understand expected signatures and method names.