PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillScriptBench: Benchmarking Self-Evolution of Executable Agent Skill Packages Beyond Markdown

Yuxuan Liu, Haoran Li, Yuhao Zhang, Jiahe Guo, Hongyu Luo, Wenbin Hu, Huihao Jing, Kawai Chung, Junle Chen, Changxuan Fan, Qing Zong, Lingyun Xie, Yangqiu Song

arXiv:2610.04008v2cs.AIcs.LGcs.SE

TL;DR

Existing benchmarks do not separately evaluate documentation repair, script repair, and preservation in executable Agent Skills. SkillScriptBench evaluates these capabilities, and AST-Guided Skill Revision links requirements to code structure to coordinate targeted script and documentation edits. Across four LLMs, the revision stage improves faulty-package repair success and repeated-run consistency.

  • Problem

    Existing benchmarks do not systematically distinguish documentation repair, script repair, and preservation when evaluating executable skill self-evolution.

    Source: §1 · p. 2

  • Method

    SkillScriptBench evaluates executable skill maintenance across in-the-wild and controlled tasks, while AST-Guided Skill Revision uses structural feedback to target code edits and align documentation.

    Source: §1 · pp. 2–3

  • Results

    Across four LLMs, AST-Guided Skill Revision raises faulty-package repair success by 21.9 percentage points for Raw Package and 27.7 for CoEvoSkills.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AST-Guided Skill Revision improves repair and preservation on Raw Package and CoEvoSkills outputs across the evaluated artifact states.

    Source: §7 · p. 9

    The In-the-Wild track is predominantly Python, both tracks use constructed faults, and long-horizon downstream Agent workflow reliability remains unevaluated.

    Source: D LIMITATIONS · p. 31

Abstract

from arXiv · show

Executable Agent Skills combine natural-language instructions and scripts into reusable packages for LLM agents, and revising them requires fixing errors without breaking correct behavior. Existing benchmarks do not systematically distinguish documentation repair, script repair, and preservation when evaluating skill self-evolution. We introduce SkillScriptBench, a 350-task benchmark designed to evaluate these capabilities separately. From a survey of over 35,000 GitHub-hosted Skill roots, we select 100 packages and construct 150 repair tasks. Each task pairs a package containing injected script faults with a maintenance request and executable checks of the required behavior. A complementary controlled track contains 200 tasks from 50 packages, each evaluated under the same maintenance request in four states: clean, documentation faults, script faults, and faults in both. Across four LLMs, methods that edit both documentation and scripts can repair script faults but do not consistently outperform Markdown-only revision on documentation repair or preservation. We therefore introduce AST-Guided Skill Revision, which uses abstract syntax trees and calling relationships to link maintenance requirements to relevant code locations. It restricts script edits to these locations and updates the documentation to match the revised scripts. Averaged across models, this revision stage yields absolute gains in repair success of 21.9% for Raw Package and 27.7% for CoEvoSkills on faulty packages. Absolute gains in the proportion of tasks solved in all three runs reach 20.8% and 31.5%, respectively, indicating more consistent repair success across repeated runs.

1 INTRODUCTION

Executable Agent Skills pair reusable instructions with scripts, so self-evolution must repair both while preserving correct behavior. SkillScriptBench evaluates these needs, and AST-Guided Skill Revision adds structure-guided coordination to existing revision methods.

1 INTRODUCTION

  • Existing benchmarks do not distinguish documentation repair, script repair, and preservation, although executable skills can fail independently across documentation and implementation.Source: §1 · pp. 1–2
  • SKILLSCRIPTBENCH comprises 350 tasks across repository-sourced In-the-Wild Repair and controlled clean and faulty package states.The benchmark includes 150 repair tasks from 100 packages and 200 controlled tasks from 50 packages.Source: §1 · p. 2
  • 21.9 and 27.7 percentage-point gains in faulty-package repair success accompany AST-Guided Skill Revision for Raw Package and CoEvoSkills, respectively.The stage also raises the proportions solved in all three runs by 20.8 and 31.5 percentage points, respectively, and improves clean-package preservation over corresponding baselines.Source: §1 · p. 2
  • Across four LLMs, full-package revision repairs script faults but does not consistently outperform Markdown-only revision on documentation repair or clean-package preservation.Source: §1 · p. 3

2 SKILLSCRIPTBENCH

SkillScriptBench evaluates maintenance of executable Agent Skill packages through complementary in-the-wild and controlled tracks, with checks for requested behavior, documentation consistency, and preservation.

2.2 BENCHMARK CONSTRUCTION

  • The 350-task benchmark combines 150 injected-script-fault tasks from 100 packages with 200 controlled tasks from 50 packages.In the controlled track, each package and fixed request appears in clean, documentation-fault, script-fault, and joint-fault states; construction checks verify reference and fault behavior.Source: §2 · p. 3, §2.2 · pp. 3–4

2.1 TASK FORMULATION

  • Each task pairs an existing executable skill package with a maintenance request and seeks behavior that satisfies the request while preserving behavior outside its scope.Source: §2.1 · p. 3

2.3 BENCHMARK COMPOSITION

  • The In-the-Wild track spans six application domains, while Controlled Repair covers Python, JavaScript, Shell, and mixed-language packages.Source: §2.3 · p. 4
  • Controlled Repair partitions tasks into single-target, multi-target, and cross-call repair according to the script locations and dependencies involved.Source: §2.3 · p. 4

2.4 EVALUATION FRAMEWORK

  • Success requires passing applicable behavioral tests and documentation-driven tests that execute task-relevant instructions against revised scripts.For clean and documentation-fault tasks, preservation checks verify existing correct functionality.Source: §2.4 · p. 4

3 AST-GUIDED SKILL REVISION

AST-Guided Skill Revision grounds maintenance requirements in code structure, limits script edits to identified targets, and aligns documentation with the resulting implementation.

3 AST-GUIDED SKILL REVISION

  • The method analyzes the original package and an initial revision to derive repair targets and preservation constraints, then produces a final package with aligned documented usage.Source: §3 · p. 5

3.1 REQUIREMENT–STRUCTURE GROUNDING

  • Language-native AST/CST analysis binds requirements and related Markdown claims to syntax nodes, while caller–helper relationships add context across function boundaries.Source: §3.1 · p. 5

3.2 REPAIR TARGETS AND PRESERVATION CONSTRAINTS

  • Requirement-linked discrepancies become editable repair targets, while behavior assessed as correct supplies preservation constraints for subsequent script edits.The method assesses both original and initially revised packages, retaining relevant symbols and call context when the initial revision removes them.Source: §3.2 · p. 5

3.3 COORDINATED MARKDOWN–SCRIPT REVISION

  • Script replacements are restricted to selected syntax nodes, validated for syntax and applicable interface preservation, and assembled from compatible edits.The method then makes localized Markdown edits to align invocations and instructions with the request and revised scripts.Source: §3.3 · pp. 5–6

3.4 INTEGRATION WITH COEVOSKILLS

  • For CoEvoSkills integration, an independent verifier diagnoses request-linked failures, and AST-Guided Skill Revision is applied to its returned package.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments compare Markdown-only revision with methods that can edit full packages, across 350 tasks, four LLM backbones, and three runs. AST-guided revision raises repair success over full-package baselines, while documentation repair remains a weakness for those baselines.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers 350 tasks with GPT-5.5, GPT-5.6-Sol, Claude Opus 5, and DeepSeek V4 Pro.The tasks comprise 150 In-the-Wild repair cases and 200 Controlled Repair tasks.Source: §4.1 · p. 6
  • Markdown-only edits SKILL.md, while Raw Package and CoEvoSkills can edit scripts as well; AST-guided variants augment the latter two methods.CoEvoSkills uses surrogate-verifier feedback to guide package revision.Source: §4.1 · p. 7
  • Each model–method–task combination is run three times; Avg is mean success, P@3 means success in at least one run, and Hit3 means success in all three.Table 2 pools 300 faulty-package tasks for Overall; Figure 4 summarizes preservation on 50 Clean tasks.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • Raw Package and CoEvoSkills score 54.8% and 48.8% Overall Avg, but their documentation-repair scores of 27.3% and 23.2% are near Markdown-only’s 27.0%.Both full-package baselines repair script faults, joint faults, and In-the-Wild cases.Source: §4.2 · p. 7
  • 76.7% and 76.5% Overall Avg for Raw Package + AST and CoEvoSkills + AST improve over their respective baselines by 21.9 and 27.7 pp.P@3 reaches 86.1% and 85.9%, and Hit3 reaches 65.0% and 65.7%; all three metrics improve for both methods on every backbone.Source: Table 2 · p. 6, §4.2 · p. 7

5 ANALYSIS

The analysis finds that AST-guided revision can improve fault correction while preserving correct behavior, and that gains vary across task structure, repair category, language, and request localization.

5.1 RQ1: WHAT DOES AST-GUIDED REVISION ADD BEYOND FULL-PACKAGE ACCESS?

  • For Raw Package documentation-fault tasks, AST-guided revision changes 42.8% of task/run pairs from failure to success and only 0.8% from success to failure.On Clean tasks, mean Hit3 rises from 85.5% to 96.5% for Raw Package and from 93.0% to 96.5% for CoEvoSkills.Source: Figure 4 · p. 7, §5.1 · p. 8
  • Without AST-guided revision, Raw Package and CoEvoSkills have lower Clean-task Avg than Markdown-only by mean gaps of 7.8 and 3.7 percentage points.The ablation’s full configuration scores 84.8% Avg, compared with 69.2% without Structural Feedback and 67.3% without Call Context.Source: §5.1 · pp. 7–8

5.2 RQ2: HOW DO REVISION GAINS VARY ACROSS TASK CHARACTERISTICS?

  • 32.8, 27.6, and 24.1 pp are the Avg gains on Single-Target, Multi-Target, and Cross-Call repairs; Multi-Target reaches 81.9%, while Cross-Call remains least successful at 71.9%.Multi-Target outperforms Single-Target across all four backbones.Source: Figure 5 · p. 8, §5.2 · p. 8
  • Argument-binding Avg rises from 50.5% to 83.3%, and helper/recovery from 44.5% to 70.4%; branch/route rises from 27.8% to 30.6%, while multi-file pipeline stays at 61.1%.Schema/field mapping improves from 77.3% to 83.5%.Source: Figure 5 · p. 8, §5.2 · p. 9
  • JavaScript Avg rises from 55.5% to 79.9%, while Python rises from 64.9% to 78.0%, yielding similar post-revision success rates despite JavaScript’s lower baseline.Source: Figure 5 · p. 8, §5.2 · p. 9
  • Avg improves by 13.8 pp on identifier-overlap tasks and 17.2 pp on other tasks, narrowing the cross-model mean gap from 13.4 to 10.0 points.Source: Figure 5 · p. 8, §5.2 · p. 9

6 RELATED WORK

Related work covers skill self-evolution, Agent Skill benchmarks, and structure-aware program repair. These lines of work do not systematically evaluate repair and preservation across both Markdown instructions and executable scripts.

6 RELATED WORK

  • SkillOpt and SkillAudit revise skill documents, while CoEvoSkills and SkillEvolver support package-level evolution of instructions and executable code.Source: §6 · p. 9
  • Existing Agent Skill benchmarks assess skill utility, continual skill generation, or skill-distillation pipelines, but do not systematically evaluate repair of packages containing Markdown and scripts.Source: §6 · p. 9
  • Structure-aware program repair localizes code changes, while comment-updating work aligns natural-language comments with code; these settings do not address joint Markdown–script revision for consistency.Source: §6 · p. 9

7 CONCLUSION

SkillScriptBench evaluates repair and preservation across executable Agent Skill packages, while AST-Guided Skill Revision coordinates Markdown and script changes through requirement-grounded structural feedback. The conclusion reports improved repair, preservation, and mean all-three-run success when this stage is applied to Raw Package and CoEvoSkills.

7 CONCLUSION

  • AST-Guided Skill Revision improves repair and preservation and raises mean all-three-run success across all four artifact states when applied to Raw Package and CoEvoSkills.The benchmark distinguishes fault repair from preserving correct package behavior.Source: §7 · p. 9

A.1 BENCHMARK COMPOSITION AND CONSTRUCTION CHECKS

  • SkillScriptBench is a 350-task benchmark combining In-the-Wild Repair built from a scan of over 35,000 public Skill roots with Controlled Repair using 50 canonical packages across four matched artifact states.Source: §7 · p. 9, §A.1 · p. 12, Table 4 · p. 12

A.2 EVALUATION DETAILS

  • Acceptance checks exercise executable behavior and documented workflows, while Clean tasks and reference invocations test preservation of correct behavior.Behavioral probes cover functions, script entry points, caller–helper workflows, or repository tests; documentation-driven checks execute commands from revised SKILL.md files.Source: §A.2 · p. 14

A.3 REQUIREMENT FEEDBACK AND STRUCTURAL BINDING

  • Requirement records link requested behavior and code assessments to editable locations; satisfied code requirements can impose preservation constraints, while uncertain assessments authorize neither edits nor constraints.A script target requires a high-confidence requirement, a REPAIR assessment, and a uniquely resolved EDIT binding.Source: §A.3 · pp. 15–16

A.4 UNIFIED REVISION AND PACKAGE ASSEMBLY

  • AST-guided revision discovers requirements for the original package and initial revision, edits admitted script targets, assembles validated changes, then aligns SKILL.md with the request and assembled implementation.Final checks restrict changed paths to SKILL.md and files under scripts/.Source: §A.4 · pp. 16–17

A.5 COMPARED METHODS AND ADDITIONAL RESULTS

This appendix describes the compared revision methods and additional evaluations. It also specifies an ablation study of structural feedback and caller–helper context on Controlled Repair.

A.5 COMPARED METHODS AND ADDITIONAL RESULTS

  • Full-package editing is restricted to SKILL.md and scripts/, while Markdown-only revision follows its initial patch with one feedback-based revision.The CoEvoSkills adaptation uses a surrogate verifier and can trigger one generator revision when the verifier or executed checks return FAIL.Source: §A.5 · pp. 17–18
  • Additional analyses report Pass@3 on Clean tasks, Controlled Repair, and all 350 tasks, plus paired confidence intervals for Overall repair Avg gains on 300 faulty-package tasks.The confidence intervals use 10,000 package-level bootstrap resamples, stratified by track membership.Source: §A.5 · pp. 18, 20

A.6 ABLATION STUDY

  • The Controlled Repair ablation compares Raw Package and two variants that remove structural feedback or caller–helper context, while retaining Markdown alignment.Both variants start from identical Raw Package outputs for each task and run; Table 18 reports state-wise and overall Avg across three runs.Source: §A.6 · p. 20, Table 17 · p. 20, Table 18 · p. 20

A.7.1 PERFORMANCE ACROSS TASK GROUPS

AST-guided revision improves joint behavioral and documentation acceptance on Controlled Repair, while additional results examine repair categories, language, and preservation. The reported outcomes also distinguish documentation fixes from behavior preservation.

A.7.1 PERFORMANCE ACROSS TASK GROUPS

  • Both Python and JavaScript improve on helper/recovery, CLI/path, and schema/field repairs; order/threshold improves for Python and remains unchanged for JavaScript.Figure 7 reports In-the-Wild task-group averages over three runs across all four backbones.Source: §A.7.1 · p. 21, Figure 7 · p. 21, Table 20 · p. 22

A.7.2 REPAIR AND PRESERVATION

  • Across eight model–generator settings, AST-guided revision corrects 49.4–60.7% of initially failing task/run pairs and regresses 0.9–4.7% of initially successful pairs.The rates pool three paired runs on 300 faulty-package tasks for each model and generator.Source: Figure 8 · p. 22, §A.7.2 · p. 22
  • On Doc tasks, 65.8% of Raw Package outputs and 74.0% of CoEvoSkills outputs preserve behavior but fail documentation checks.These outcomes illustrate why package success requires both executable behavior and documented use to pass.Source: §A.7.2 · pp. 22–23
  • On Doc tasks, AST-guided revision raises joint acceptance from 27.3% to 69.3% for Raw Package and from 23.2% to 70.3% for CoEvoSkills.Figure 9 separates behavioral and documentation outcomes; on Script and Joint tasks, AST-guided revision also reduces the share passing documentation checks alone.Source: §A.7.2 · pp. 22–23, Figure 9 · p. 23

A.7.3 CONSISTENCY AND TASK COMPOSITION

Matched-state results show that AST-guided revision increases consistent success across fault states and the corresponding Clean state. Composition analyses find that gains persist under alternative weighting, while language comparisons vary with category weighting.

A.7.3 CONSISTENCY AND TASK COMPOSITION

  • Complete four-state success rises from 17/200 (8.5%) to 67/200 (33.5%) for Raw Package and from 6/200 (3.0%) to 83/200 (41.5%) for CoEvoSkills with AST-guided revision.The four states are counted successful only when all three runs pass; Figure 10 gives the matched package–model counts.Source: Figure 10 · p. 23, §A.7.3 · p. 23
  • Equal category weighting changes the Python–JavaScript gap from 12.3 to 5.1 points for Raw Package and from 13.3 to 16.9 for CoEvoSkills; adjusted +AST gaps are −8.1 and 5.7 points.These comparisons use the same tasks and categories shared by each language group.Source: §A.7.3 · p. 24, Figure 11 · p. 24
  • After category adjustment, multiple-target Avg remains higher than single-target Avg by 5.2 points for Raw Package + AST and 11.7 points for CoEvoSkills + AST.The reported contrast supports the conclusion that target count alone does not order repair difficulty.Source: §A.7.3 · p. 24
  • In-the-Wild gains remain positive for every backbone under task, source-package, and category weighting: 15.2, 10.8, and 13.5 points for Raw Package, and 22.9, 19.7, and 22.2 for CoEvoSkills.Figure 11(a) compares these three weighting schemes.Source: Figure 11 · p. 24, §A.7.3 · p. 24

A.8 CASE STUDY: COORDINATED MARKDOWN–SCRIPT REPAIR

The VCP case study tests coordinated repair across documentation and scripts under a shared maintenance request, then examines how AST-guided revision changes the package and its behavior. The appendices describe the revision stages and baseline prompts.

A.8 CASE STUDY: COORDINATED MARKDOWN–SCRIPT REPAIR

  • The four package states share one request: Clean, Doc, Script, and Joint isolate documentation faults, script faults, both, or neither.The script and joint states substitute six argument values; the documentation fault changes the instruction against reusing neighboring VCP options.Source: §A.8 · p. 25, Table 22 · p. 25
  • Raw Package + AST passes all ten behavioral checks, versus four for Raw Package, and the revised package also passes documentation evaluation and detects the expected contracting patterns.On non-contracting inputs, neither package produces detections; the revised package also corrects all six argument bindings while preserving public function signatures.Source: §A.8 · pp. 25–26

B.1 REQUIREMENT DISCOVERY

  • Requirement discovery records request and document evidence, assesses discrepancies separately for parent and proposal, and proposes the smallest coherent set of code locations for repair.Location bindings are hypotheses rather than edit permissions; the host validates requested edits against parsed nodes.Source: §B.1 · pp. 26–27

B.2 SCRIPT REVISION

  • Script revision validates requirements and preservation constraints before editing, then submits either all declared-node edits or no edits while preserving public signatures and unrelated behavior.The host checks edit scope, syntax, and interface preservation.Source: §B.2 · p. 27

B.3 MARKDOWN ALIGNMENT

  • Markdown alignment reviews every numbered invocation against scripts and requires script evidence for consistency judgments; contradicted invocations must be addressed by bounded edits.The instructions prohibit inventing workflow stages or weakening public requirements, and the program validates references rather than behavioral correctness.Source: §B.3 · p. 28

C.1 MARKDOWN-ONLY

  • The Markdown-only baseline can edit visible Markdown but not executable files, and it is instructed to preserve unrelated behavior and public interfaces.Its prompt also specifies bounded line edits and zero edits when no change is justified.Source: §C.1 · p. 29

C.2 RAW PACKAGE

  • Raw Package can inspect and edit SKILL.md and existing scripts, repairing connected caller-helper and cross-file defects while preserving unrelated behavior and interfaces.Its prompt provides no AST locations, hidden tests, verifier output, or prior benchmark outcomes.Source: §C.2 · p. 30

C.3 COEVOSKILLS

  • CoEvoSkills separates immutable-package verification from a single generator refinement, treating the surrogate report as a fallible hypothesis to recheck before editing.The verifier derives checks from the public task and request, while the generator makes a smallest coherent repair or submits zero edits.Source: §C.3 · pp. 30–31

D LIMITATIONS

The evaluation is bounded by its language and fault construction, and it measures package-level correctness rather than long-horizon agent reliability.

D LIMITATIONS

  • The In-the-Wild track is predominantly Python, while both tracks use constructed faults, limiting evidence about broader languages and naturally occurring failures.The paper identifies broader language coverage and naturally occurring failures as directions for assessing generalization to the wider Skill ecosystem.Source: D LIMITATIONS · p. 31
  • The evaluation measures package-level correctness and documentation–script consistency, not reliability in long-horizon downstream Agent workflows.The paper states that long-horizon workflow reliability remains to be evaluated.Source: D LIMITATIONS · p. 31
  • The benchmark's results therefore speak to package maintenance under the evaluated tracks, not demonstrated generalization across the wider Skill ecosystem.The stated scope boundary is the predominantly Python In-the-Wild track and constructed faults in both tracks.Source: D LIMITATIONS · p. 31

Related papers

Loading 2610.04008v2…