Source-linked AI summary
SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents
Yifan Zhou, Zhentao Zhang, Ziming Cheng, Shuo Zhang, Qizhen Lan, Zhangquan Chen, Zhi Yang, QianyuXu, Ronghao Chen, Huacan Wang, Sen Hu
TL;DR
SkillGenBench addresses the lack of benchmarks that isolate whether pipelines can distill correct, reusable, executable skills from repositories and documents. It standardizes generation and fixed-harness evaluation across task-conditioned and task-agnostic settings, finding that performance depends on the method, backbone, and source, with repository tasks harder and specification-to-execution gaps persistent.
Problem
Existing benchmarks usually assess provided-skill efficacy or end-to-end task solving, leaving skill generation from raw corpora insufficiently isolated despite its importance for reusable procedural artifacts.
Method
SkillGenBench separates generation from execution, producing standardized skills from repository- or document-grounded sources under task-conditioned or task-agnostic protocols with fixed executors and deterministic checks.
Results
Skill generation performance varies with the method, backbone, and source; repository-grounded tasks are harder than document-based ones, and generated skills often fail to turn correct structure into executable procedures.
Takeaways & Limitations
Static diagnostics explain skill contents, while execution-based evaluation determines whether generated skills actually work, making both complementary for studying procedure-to-skill distillation.
Takeaways & Limitations
Evaluation assumes hidden test cases, verifier internals, reference outputs, and held-out tasks remain unavailable during generation, with all skills executed in containerized environments under one executor.
Abstract
from arXiv · showhide
As LLM agents are increasingly built around reusable skills, a central challenge is no longer only whether agents can use provided skills, but whether they can generate correct, reusable, and executable skills from repositories and documents. Existing benchmarks primarily evaluate the efficacy of given skills or the ability of agents to solve downstream tasks from raw context, but they do not isolate skill generation itself as the object of study. We introduce SkillGenBench, a benchmark for evaluating skill generation pipelines under a unified and controlled protocol. In SkillGenBench, a generator receives raw corpora and produces standardized skill artifacts, which are then executed under fixed harnesses and assessed with unified evaluation procedures. The benchmark covers two generation regimes: task-conditioned generation, where a task-specific skill is synthesized after the task is revealed, and task-agnostic generation, where a reusable skill library must be distilled before downstream tasks are known. It also spans two complementary procedural sources: repository-grounded instances, where procedures are distributed across code, configuration, and scripts, and document-grounded instances, where procedures and constraints must be distilled from long-form text. We provide standardized task specifications, pinned environments, and evaluation protocols centered on deterministic execution-based checks, supplemented by auxiliary signals for diagnosis. Experiments across a range of skill-generation methods and backbones show substantial performance variation, highlight the difficulty of reusable skill distillation, and reveal distinct failure modes in skill generation from software repositories versus long-form documents. SkillGenBench establishes a reproducible testbed for studying skill generation as an independent research problem in agent systems.
1 Introduction
SkillGenBench targets the under-evaluated problem of generating correct, reusable, and executable skills from raw repositories and documents. It evaluates standardized generated artifacts across task-conditioned and task-agnostic settings with controlled execution and diagnostic protocols.
- Existing benchmarks usually evaluate provided-skill efficacy or end-to-end task solving, rather than isolating skill generation from raw corpora.
- SkillGenBench gives a generator raw corpora and evaluates the standardized skill artifacts it produces under fixed harnesses and unified procedures.
- Task-conditioned generation uses the revealed task to synthesize a focused skill, whereas task-agnostic generation distills a reusable library before downstream tasks are known.
- The benchmark spans repository-grounded and document-grounded procedural sources, with standardized specifications, pinned environments, and deterministic execution checks supplemented by diagnostic signals.
2 Related Work
Related work extends agents through runtime augmentation and reusable skills, but existing benchmarks mainly test whether supplied or self-generated skills improve downstream execution. SkillGenBench instead evaluates the skill-generation process itself.
- Runtime augmentation improves single-episode capability through reasoning loops, tools, retrieval, and interfaces, while leaving procedural knowledge implicit in prompts, traces, or retrieved context.
- Skill research increasingly treats procedures as reusable artifacts, spanning experience-driven consolidation, corpus distillation, explicit packaging, orchestration, and iterative refinement.
- Existing skill benchmarks primarily assess whether a provided skill improves downstream execution, including comparisons among no-skill, curated-skill, and self-generated settings.
3 SkillGenBench
SkillGenBench decouples skill generation from downstream execution in a containerized benchmark spanning repository- and document-grounded procedures, two task-knowledge regimes, and programmatically verified tasks. Its construction and evaluation pipelines combine controlled task validation with hidden execution checks and auxiliary diagnostics.
- 3.3 Benchmark Construction: The construction pipeline abstracts sources into knowledge graphs, generates and filters scenarios and test cases, verifies difficulty with and without skills, and human-checks accepted tasks.
- 3 SkillGenBench: SkillGenBench treats skill generation as the primary object of evaluation by separating source analysis and skill creation from execution by a separate executor.
- 3 SkillGenBench: Each instance packages source materials, task specification, skill interface, executor, and evaluation protocol in a containerized environment.
- 3.1 Sources of Procedural Knowledge: Repository-grounded instances require recovering implicit workflows from code structure, call relations, scripts, configurations, and runtime constraints.
- 3.1 Sources of Procedural Knowledge: Document-grounded instances require integrating explicit but dispersed conditions, parameter rules, prerequisites, and ordered steps across long-form sources.
- 3.2 Task Settings: Task-conditioned generation targets procedures relevant to a revealed task, whereas task-agnostic generation builds a reusable library for held-out tasks without task hindsight.
- 3.4 Evaluation Protocol: Evaluation runs generated skills in the same containerized executor, using hidden deterministic tests for callable procedures or reusable implementations.
4 Experiments
Experiments evaluate skill-generation pipelines across methods, backbones, source types, and generation regimes using execution results, static diagnostics, and verifier-failure analysis. Results show strong dependence on pipeline components and source structure, with repository tasks and task-agnostic generation remaining especially challenging.
- Dynamic execution results: SKILLSEEKERS achieves the best average performance across six generation backbones, reaching 14.4% on Code and 25.0% on Doc.Performance improvements are not stable and depend on the interaction between generator, backbone model, and source type.
- Dynamic execution results: Generated skills can underperform no-skill baselines when they introduce interface inconsistencies, incomplete procedures, or incorrect assumptions.Skills help most when they provide precise, source-grounded procedures that the executor cannot easily infer.
- Dynamic execution results: Code performance remains 10.8%–14.4%, whereas Doc performance reaches 21.4%–25.0% across methods.Repository tasks require recovering implicit execution structure, including environment setup, command conventions, and data flow, from distributed artifacts.
- Generation regimes: Task-agnostic generation often performs worse than task-conditioned generation and can underperform no-skill baselines because reusable skills omit precise downstream constraints.Increasing the generation budget helps only up to roughly 24K–64K tokens, after which gains saturate.
- Static diagnostics: Static diagnostics and execution results measure different aspects of quality: structural completeness does not guarantee executability, and dynamic success does not guarantee sound structure.SKILLNET leads grouped static scores, while SKILLSEEKERS has the strongest Code and Doc averages, best Safety score, and strong Grounding.
- Failure analysis: Verifier failures differ by source: Code Repo failures are dominated by runtime or dependency issues, while Code Doc failures are dominated by interface or schema errors.The reported Code Repo counts are 1245 runtime/dependency failures (53%), 626 interface/schema failures (27%), and 475 asset/artifact failures (20%); Code Doc has 450 interface/schema failures (85%).
- Conclusions from experiments: Overall, skill generation should be evaluated as a generator–backbone–executor pipeline because repository, code-documentation, and domain-document tasks require different kinds of procedural fidelity.The results identify operational recovery, exact interface compliance, and precise rule execution as source-specific challenges.
5 Conclusion
SkillGenBench treats skill generation as a first-class problem and enables controlled comparison across repository and document sources. Its experiments show that performance depends on the generator, backbone, and source material, while execution-level correctness remains distinct from structural completeness.
- Contribution: SkillGenBench establishes skill generation as a first-class problem by decoupling upstream skill generation from downstream execution.The benchmark supports controlled comparison of procedure-to-skill distillation pipelines across repository and document sources.
- Main findings: Performance depends on the generation method, backbone model, and source material, with repository-grounded tasks more challenging than document-based tasks.The difficulty reflects the need to recover implicit execution structure from distributed code artifacts.
- Implications: Static diagnostics explain what a skill contains, whereas execution-based evaluation determines whether it works under strict verification constraints.The persistent gap is especially pronounced for interface alignment, state handling, and rule fidelity.
- Implications: SkillGenBench provides a benchmark and analysis framework for improving executable, reliable, and procedurally aligned skill artifacts.The stated future direction is to address execution-level correctness beyond surface-level structure.
A.1 Method–Backbone Heatmap
The method–backbone matrix shows that downstream execution is fixed while skill-generation methods and backbones vary, revealing backbone-dependent performance and no uniformly dominant method. Pass rates rise with generation budgets before reaching backbone-dependent plateaus.
- Method–Backbone Heatmap: No method dominates uniformly across generation backbones.SKILLSEEKERS leads on four backbones, SKILLNET reaches 20.3% on Qwen3.6 Plus, and SKILLCREATOR is near SKILLSEEKERS on GLM-5.
- Method–Backbone Heatmap: SKILLSEEKERS remains within a 14.4–20.9% pass@3 range across backbones, whereas EVOSKILL spans 10.2–20.3%.The wider EVOSKILL spread indicates greater sensitivity to backbone choice than SKILLSEEKERS.
- Token-Budget Sensitivity: Pass rate rises steeply through roughly 16K–24K generation tokens, then flattens between 32K and 64K.Increasing the budget to 96K or 128K adds little further gain on the 187-task suite.
- Token-Budget Sensitivity: Plateau heights depend on the backbone: GPT-5 and GLM-5 approach 18–20% pass@3, while Kimi K2.5 and MiniMax M2.7 reach 10–17%.The backbone-dependent plateaus show that additional generation budget alone does not close the remaining performance gap.
A.3 Bootstrap Confidence Intervals
The benchmark reports task-level bootstrap confidence intervals for overall pass@3 while fixing the evaluation harness, runtime, environment, and executor-side settings. The intervals are broad enough that most pairwise method differences are not statistically distinguishable at this scale.
- Bootstrap Confidence Intervals: B=2000 task-level bootstrap resamples over 187 tasks produce confidence-interval half-widths of approximately ±5 percentage points.Each Table 4 cell reports a mean with its bootstrap 95% confidence interval.
- Interpretation: Most pairwise method differences are not statistically distinguishable at the benchmark’s 187-task scale.The reported uncertainty supports considering method and backbone choices jointly rather than treating small gaps as decisive.
- Evaluation Setup: All experiments use the same SkillGenBench harness and Claude Code runtime, with downstream execution fixed to MiniMax-2.5.The runtime uses Claude Code CLI 2.1.85 and claude-agent-sdk 0.1.64.
- Evaluation Setup: Downstream evaluations run in isolated Docker environments selected by instance configuration without requested GPU resources.The generation-stage configuration allows 16,384 output tokens and up to three refinement iterations or 45 agent turns.
C Baseline Methods
The baselines compare prompt-based, workflow-based, and self-evolving procedures for constructing standardized skills under a shared visibility boundary. Case studies show that repository- and document-specific procedural details can determine whether outputs are executable and numerically correct.
- Baseline Families: Five baselines span prompt-based, workflow-based, and self-evolving skill-generation families under a shared benchmark interface.All methods output the same SKILL.md package, focusing comparison on skill construction rather than downstream execution.
- Prompt-Based Generation: NAIVE PROMPT directly writes a skill package from the visible corpus, without trajectories, search, self-evaluation, or an explicit authoring workflow.In task-conditioned generation, it also receives the task instruction.
- Workflow and Self-Evolving Generation: SKILLNET and SKILLSEEKERS use explicit source-to-skill workflows, while EVOSKILL derives skills from execution trajectories.SKILLNET represents toolkit-mediated creation; SKILLSEEKERS emphasizes extracting and packaging actionable knowledge from repositories and documents.
- Repository-Grounded Case Studies: Repository cases require implementation-specific details such as CLI conventions, BGR conversion, deterministic seeding, and unique username generation.These constraints govern valid outputs in StyleTransfer, AnimeGANv3, and Faker tasks.
- Document-Grounded Case Study: The PDFPlumber case requires page.extract_words() for word counts and default-layout page.extract_text() with newline splitting for line counts.Using whitespace splitting, layout-preserved extraction, or inferred y-coordinate lines can produce structurally valid but numerically incorrect JSON.
E Limitations
SkillGenBench is a controlled benchmark rather than a complete representation of agent-skill deployment. Its scope is limited by backbone coverage, diagnostic failure analysis, deterministic fixed execution, incomplete source domains, proxy scores, and broader automation risks.
- Scope: Current dynamic execution results cover six generation backbones, with future releases expected to provide self-contained raw run directories for every summary row.This bounds the benchmark’s current empirical coverage.
- Failure Analysis: The completed-failure taxonomy supports large-scale diagnosis but does not replace human adjudication because individual failures may have overlapping causes.The taxonomy combines execution traces, generated code, and task metadata.
- Deployment Scope: Deterministic verifiers and fixed downstream execution under-represent interactive settings involving user negotiation, external services, or post-deployment skill revision.The benchmark also omits broader domains, larger repositories, multi-repository workflows, and longer task-agnostic libraries.
- Measurement: Static scores are rule-based diagnostics rather than intrinsic measures of skill quality.The paper presents them as useful for explaining observed failures but not as complete quality measures.
- Broader Impact: Generated workflows may lower the barrier to executing complex tasks, creating misuse risks that require careful evaluation and monitoring.The benchmark’s broader impact includes both potential reliability benefits and automation-related risks.
F.1 Human Verification
Benchmark construction combines automatic screening with manual review, while the generation pipeline builds knowledge graphs, summarizes communities, and produces computation-bearing scenarios and executable task bundles. Manual verification retains only candidates meeting all five criteria.
- F.1 Human Verification: Automatic checks remove candidate tasks that are too easy, brittle, or unlikely to support stable verification, followed by manual review for clarity and alignment.The manual pass complements automatic filtering by checking clarity, coverage, and procedural-recovery alignment.
- F.1 Human Verification: 678 candidate tasks were manually inspected, and 187 passed all five criteria, yielding a 27.6% acceptance rate.Repairable failures returned to refinement; other failed candidates were discarded.
- Knowledge Graph Summary: Community detection condenses the merged knowledge graph into short thematic summaries that provide context for downstream scenario generation.Each community is summarized around the topic or theme represented by its related entities and relationships.
- Knowledge Graph Construction: The knowledge-graph stage proposes entity types, extracts entities, and emits subject–predicate–object relations in one structured JSON output.Entities are normalized and selected for substantive mentions and relationship potential, while relations must be explicit or clearly implied with correct directionality.
- Scenario Generation: Scenario generation targets realistic tools or systems that integrate multiple document sections and require computation, logic, or decision-making.The prompt excludes simple lookups, exam-style questions, common-knowledge scenarios, and overly narrow or abstract tasks.
- Task and Test-Case Generation: Task generation jointly creates a general function interface and executable test cases whose solution code contains document-specific constants.This design enforces a contamination boundary between abstract task descriptions and concrete test-case solutions.