PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study of Agent Skills' Downstream Utility

Yu Cheng, Dehai Zhao, Zhongxin Liu, Qing Huang, Zhenchang Xing, Xiaoxue Ren

arXiv:2610.08875v2cs.AIcs.LG

TL;DR

Relevant Skills do not reliably improve task performance, and prior studies leave limited explanation of how utility depends on Skill content, execution configuration, and multi-Skill organization. The study compares Skills across configurations, alternative candidates, and fixed-set organizations while analyzing content, traces, and artifacts. Utility varies across configurations, and operation-support reranking and explicit organization schemes improve evaluated outcomes.

  • Problem

    Prior work characterizes Skill content and measures task performance but provides limited explanation of how utility depends on content, model–harness configuration, and multi-Skill organization.

    Source: §1 · p. 1

  • Method

    Across 87 tasks, the study compares Skills across configurations, alternatives, and fixed-set organizations, then analyzes content, execution traces, and final artifacts with author review.

    Source: §1 · p. 2, §3.5.3 · p. 6

  • Results

    36.78% of tasks show the same Skills helping some configurations and hurting others; operation-support reranking raises first-choice pass rates by 4.35–5.80 percentage points across three configurations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Developers can assess usable operation support, adapt procedures while preserving task requirements, and make artifact dependencies explicit when coordinating Skills.

    Source: §1 · p. 2

    Utility measures task completion, not partial completion or execution cost, and LLM-based operation judgments may contain annotation errors that affect rankings.

    Source: §6 · p. 11

Abstract

from arXiv · show

Agent Skills package procedural guidance and resources for reuse, but a relevant Skill does not necessarily improve task performance. Existing studies characterize Skill content and evaluate downstream performance, yet provide limited explanations of how utility depends on content, execution configuration, and multi-Skill organization. We conduct an empirical study on 87 SkillsBench tasks, defining downstream utility as the pass-rate difference from No-Skill on the same tasks under the same model--harness configuration. We compare the same Skills across nine configurations, then examine alternative published Skills and organizations of fixed Skill sets under three selected configurations. We retrieve marketplace candidates from a curated corpus of 37,596 Skills. LLM-assisted analysis of content, execution traces, and final artifacts, followed by author review, relates provided support to actual use and task outcomes. The same Skills help some configurations and hurt others on 36.78\% of tasks, with trajectories showing that recommended procedures can become an execution burden. Relevance rankings overlook more useful candidates. Within the evaluated candidate sets, reranking by support for required operations raises first-choice pass rates by 4.35--5.80 percentage points across the three configurations. We derive 17 authoring practices linking executable procedures to recovery, preservation of task requirements, and checks on final artifacts. Stage Plan and Dependency DAG outperform use order alone, with DAG's additional benefits concentrated in tasks supplied with five or six Skills. These findings guide developers to assess usable operation support, allow procedure adaptation while preserving task requirements, and make artifact dependencies explicit when organizing Skills.

1 Introduction

Relevant Skills do not reliably improve performance, and prior content and benchmark studies provide limited explanation of how utility depends on content, execution configuration, and multi-Skill organization. This study compares Skills and their use across configurations, candidate alternatives, and organization schemes, finding that utility can reverse across configurations and that operation support and explicit artifact dependencies inform selection and coordination.

1 Introduction

  • Downstream utility is the task pass-rate difference from No-Skill on the same tasks under the same model–harness configuration, and relevance alone does not guarantee improvement.Skills with similar purposes can differ in supported operations, resources, and procedural requirements.Source: §1 · p. 1
  • 36.78% of tasks show the same Skills helping some model–harness configurations and hurting others, while recommended procedures can burden execution when agents struggle to implement or adapt them.The study compares the same Skills across nine configurations.Source: §1 · p. 2
  • Reranking relevant candidates by operation support raises first-choice pass rates by 4.35–5.80 percentage points across three configurations.Lower-ranked candidates outperform the first choice in 23.19% of task–configuration groups.Source: §1 · p. 2
  • Stage Plan and Dependency DAG yield greater downstream utility than Sequence across all three configurations; DAG’s additional gains reach 3.03–12.12 percentage points on tasks supplied with five or six Skills.Intermediate artifacts support source checking and propagation of corrections.Source: §1 · p. 2
  • Developers can assess usable operation support, adapt procedures while preserving task requirements, and coordinate Skills through explicit artifact dependencies.The authors also identify needs for tools that track unsupported operations, configuration suitability, and effects of artifact changes.Source: §1 · p. 2
  • The study combines comparisons of published Skills with analysis of content, execution traces, and final artifacts to relate supplied support to actual use and outcomes.It derives 17 authoring practices, including preserving justified acceptance criteria when changing methods and specifying which changes require rechecking final artifacts.Source: §1 · p. 2

2 Background and Related Work

Agent Skills provide procedural guidance and resources for agents, but their utility depends on how models and harnesses execute that guidance. Prior work addresses Skill retrieval, organization, content, and performance, while this study compares alternatives and traces actual use to relate content to task outcomes.

2.1 LLM Agents and Agent Skills

  • A model–harness configuration pairs a model with an execution harness, whose tools and control mechanisms shape how the model carries out Skills.Agent Skills package procedural instructions and optional scripts, references, and assets; progressive disclosure exposes descriptions for discovery, bodies on activation, and resources as needed.Source: §2.1 · p. 3

2.2 Skill Retrieval and Organization

  • Prior retrieval work improves matching, but relevant Skill retrieval does not ensure appropriate loading or use; organization research motivates explicit dependencies, which this study compares while holding Skills and resources fixed.The compared schemes provide no organization instructions, a use sequence, stage objectives, or artifact dependencies; trajectories track how intermediate results are passed, checked, and revised.Source: §2.2 · p. 3

2.3 Empirical Studies of Agent Skills

  • Content studies characterize Skill components and modifications, while performance evaluations measure outcomes or obstacles; this study compares published alternatives and traces procedures and resources used, skipped, or supplemented.Related approaches include operation and execution representations [16] [13] [39] and evaluations of Skill use across harnesses [12].Source: §2.3 · p. 3, §2.2 · p. 3

3 Methodology

The methodology links task requirements, Skill content, observed execution, and outcomes through three complementary research questions. It uses a curated Skill corpus and benchmark task subsets suited to configuration, candidate, and organization comparisons.

3 Methodology

  • The study combines controlled execution with content and trajectory analysis: RQ1 compares the same Skills across configurations, RQ2 compares alternative Skills for a task, and RQ3 compares organizations of the same Skill set.The analyses connect task requirements, content, observed use, and outcomes.Source: §3 · p. 4

3.1 Corpus Construction

  • The curated corpus retains 37,596 Skills, and the benchmark comprises 87 tasks with 232 supplied Skills across eight domains.The corpus was filtered for completeness, adoption, language, and duplication; tasks were collected as of September 20, 2026.Source: §3.1 · p. 4
  • RQ1 uses all 87 tasks, RQ2 uses the 23 single-Skill tasks, and RQ3 uses the 41 tasks supplied with at least three Skills.These subsets support configuration, alternative-candidate, and organization comparisons, respectively.Source: §3.1 · p. 5

3.2 Controlled Execution

Controlled execution compares Skill and No-Skill conditions across three models and three harnesses, with repeated runs in fresh environments.

3.2 Controlled Execution

  • Three models run with Codex, OpenCode, or OpenClaw; each task condition is repeated three times in a fresh environment, and Skill runs explicitly ask agents to consult and use the supplied Skills.No-Skill runs provide neither Skills nor the corresponding consultation request.Source: §3.2 · p. 5

3.3 Utility Evidence

The section measures performance by averaging task-level pass outcomes equally, then defines Skill utility as the pass-rate change from No-Skill under the same task set and configuration. It also examines trajectories and final artifacts to connect Skill use with outcomes.

3.3 Utility Evidence

  • Task performance is the equally weighted average of per-task pass fractions, after averaging repeated outcomes within each task.A task passes when its original verifier accepts the final state.Source: §3.3 · p. 5
  • Downstream utility is the pass-rate change from No-Skill on the same task set and model–harness configuration.Source: §3.3 · p. 5
  • Execution evidence compares trajectories and final artifacts to relate actual Skill use to task outcomes across conditions.Source: §3.3 · p. 5

3.4 RQ1: Skill Utility across Model–Harness Configurations

RQ1 compares the same benchmark-supplied Skills and resources with No-Skill across nine model–harness configurations and uses the results to select configurations for later analyses.

3.4 RQ1: Skill Utility across Model–Harness Configurations

  • The study crosses three LLMs with three harnesses and evaluates all nine configurations on the 87 tasks.Source: §3.4 · p. 5
  • Each configuration compares Benchmark-Skill and No-Skill conditions, with the same benchmark Skills and resources supplied for a given task across configurations.Source: §3.4 · p. 5
  • The analysis compares condition pass rates and gains across models and harnesses, classifies task-level gain and loss patterns, and examines trajectories for tasks with mixed outcomes.Source: §3.4 · p. 5
  • The harness with the highest Benchmark-Skill pass rate is selected for each model, yielding three configurations for RQ2 and RQ3.Source: §3.4 · p. 5

3.5 RQ2: Utility Differences among Skills Targeting the Same Capability

RQ2 compares alternative Skills for the same task using execution and content evidence, assessing whether their support for required operations corresponds to utility and observed outcomes.

3.5 RQ2: Utility Differences among Skills Targeting the Same Capability

  • RQ2 compares Skill utility and support on 23 single-Skill tasks under three selected configurations, reducing variation from multi-Skill coordination.Source: §3.5 · p. 5

3.5.1 Alternative Skill Selection.

  • Top-five marketplace candidates are selected by relevance, but higher relevance ranks do not necessarily yield higher downstream utility.Candidates for a task can differ in operation coverage, inputs and outputs, and applicability conditions.Source: §3.5.1 · p. 5, §3.5.2 · p. 6

3.5.2 Atomic-operation Reranking.

  • Task and Skill procedures are decomposed into atomic operations, and candidate support is scored as full, partial, absent, or conflicting before candidates are reranked.Full support requires only task-parameter binding; partial support needs additional substantive logic, absent means no support evidence, and conflicting support violates an explicit constraint without a compatible alternative.Source: §3.5.2 · p. 6

3.5.3 Utility-Grounded Authoring Practices.

  • Authoring analysis combines candidate pass-rate contrasts with content, execution trajectories, and deliverables, then uses LLM induction and author review to derive requirements across five stages of Skill use.Authors checked evidence, considered alternative explanations and counterexamples, and distinguished recurring observations from single-case insights.Source: §3.5.3 · p. 6

3.6 RQ3: Effects of Multi-Skill Organization on Utility

RQ3 holds the Skill set and resources fixed while comparing organization instructions across 41 multi-Skill tasks in three configurations. It evaluates pass rates, task-level gains, and how intermediate artifacts are used.

3.6 RQ3: Effects of Multi-Skill Organization on Utility

  • The comparison uses 41 tasks with at least three Benchmark Skills under three selected configurations, retaining the complete Skill set, original content, and resources.Source: §3.6 · p. 7
  • Flat gives no task-specific organization, Sequence specifies use order, Stage Plan adds stage objectives, and Dependency DAG specifies prerequisite links and artifacts passed between Skill uses.DAG nodes represent individual Skill uses, and later nodes run only when required artifacts are available.Source: §3.6 · p. 7
  • The study measures organization gains against Flat and compares outcomes by number of supplied Skills, while execution analysis checks order-following, stage objectives, and intermediate-artifact use.Source: §3.6 · p. 7

4 Results

Benchmark Skills improve overall pass rates, but their task-level utility varies across configurations and candidates. The results also show that operation support, authoring practices, and multi-Skill organization relate to downstream utility.

4.1 RQ1: Skill Utility across Model–Harness Configurations

  • Benchmark Skills raise pass rates by 4.60–19.54 percentage points over No-Skill in all nine configurations, with GPT gaining more than Qwen under each harness.The table reports pass rates and gains over No-Skill across model–harness configurations.Source: §4.1 · p. 7, Table 3 · p. 7
  • In 32 of 87 tasks (36.78%), the same Skills help one configuration and hurt another; recommended procedures can burden agents unable to execute or adapt them.Figure 2 groups tasks by gains and losses across configurations; trajectories illustrate the execution burden of procedures that agents struggle to carry out.Source: §4.1 · p. 7, Figure 2 · p. 8

4.2.1 Utility Differences among Alternative Skills.

  • Across 69 task-configuration comparisons, marketplace-candidate pass rates varied in 27 (39.13%), while M1 trailed a lower-ranked candidate in 16 (23.19%).Under Qwen + OpenCode, only M2 exceeds No-Skill; M1, M3, and M4 fall below it, while M5 ties it.Source: §4.2.1 · p. 8, Table 4 · p. 8

4.2.2 Utility Gains from Atomic-operation Reranking.

  • Reranking by atomic-operation support raises first-choice pass rates over M1 by 4.35, 4.35, and 5.80 percentage points across GPT + Codex, DeepSeek + OpenClaw, and Qwen + OpenCode.For Qwen + OpenCode, the first-choice pass rate rises from 14.49% to 20.29%, changing a loss relative to No-Skill into a gain.Source: §4.2.2 · p. 8, Table 5 · p. 8

4.2.3 Utility-Grounded Authoring Practices.

  • The 17 authoring practices link executable procedures to recovery, final-deliverable checks, rechecking after changes, and preserving justified acceptance criteria during repair.The practices span five stages and include applicability, adaptation, implementation, verification, and correction requirements.Source: §4.2.3 · pp. 9–10, Table 6 · p. 9

4.3 RQ3: Effects of Multi-Skill Organization on Utility

  • Stage Plan and Dependency DAG outperform Sequence; DAG gains 6.50, 11.38, and 6.50 percentage points over Flat and achieves the highest or tied-highest overall pass rate.All conditions provide the same Skills and supporting resources.Source: §4.3 · p. 10, Table 7 · p. 10
  • DAG’s additional gains over Stage Plan concentrate in tasks with five or six Skills: 12.12, 3.03, and 9.09 percentage points across configurations.The schemes are identical in the four-Skill group, and differ by at most 1.67 percentage points in the three-Skill group.Source: §4.3 · p. 10
  • For jpg-ocr-stat, GPT + Codex passes 3/3 DAG runs versus 0/3 under Flat, Sequence, and Stage Plan; saved OCR text and field files support checks and revisions carried into the workbook.The intermediate artifacts connect cross-Skill result transfer with correction before delivery.Source: §4.3 · p. 10

5 Discussion

The study addresses its challenges by comparing Skills and analyzing their actual use, then draws implications for Skill reuse, harness design, and future evaluation. These implications connect operation support and artifact dependencies to how Skills are selected, adapted, and organized.

5.1 Addressing the Study Challenges

  • Comparisons of real-world alternatives and analysis of traces and artifacts ground the study’s interpretations, while individual content requirements still need controlled validation.The comparisons examine Skills across configurations and organization strategies, distinguishing guidance followed, skipped, or adapted from operations agents supply themselves.Source: §5.1 · p. 10

5.2 Implications for Skill Development and Research

  • Authors should specify which methods can change, which task requirements remain binding, and which final-artifact checks must be repeated after revisions.Candidate descriptions and representative regression tasks can help adopters judge support and whether reported gains apply to their settings.Source: §5.2 · p. 10
  • Harnesses could track prerequisites, guidance use, stage objectives, artifact dependencies, and downstream checks affected by upstream changes.The study calls for evaluating the correctness and cost of automatically generated organization instructions, alongside further tests across tasks, configurations, and Skill combinations.Source: §5.2 · p. 11

6 Threats to Validity

The study defines validity boundaries in its outcome measure, comparisons, sampled Skills and tasks, and repeated-run design. These boundaries constrain how its utility and reranking findings should be interpreted.

Construct Validity

  • Pass-rate utility measures task completion, leaving partial completion and execution cost outside its scope.LLM-based operation extraction and heuristic support scores may introduce errors or change rankings under different scoring choices or operation granularities.Source: §6 · p. 11

Internal Validity

  • RQ2 candidate attributes may vary together, so observed authoring features are associations rather than isolated effects.RQ3’s order, stage objectives, and artifact handoffs involve author choices, so gains are interpreted for complete organization schemes.Source: §6 · p. 11

External Validity

  • The study covers 87 tasks across eight domains and English-language Skills with complete content and adoption evidence; other languages and open-ended workflows need further study.RQ2 and RQ3 use 23 and 41 tasks, respectively, and RQ2 restricts candidates to the top five retrieved per task.Source: §6 · p. 11

Conclusion Validity

  • Three repetitions per task and condition leave task-level pass rates with limited resolution and sensitivity to execution randomness.Reranking gains require independent-task evaluation, while five- and six-Skill groups contain only six and five tasks, respectively.Source: §6 · p. 12

7 Conclusion

The study links Skill content, actual use, and task outcomes across controlled executions, content comparisons, and trajectory analysis. It finds configuration-dependent utility, improved candidate selection through operation-support reranking, and higher overall pass rates for stage-objective or dependency schemes than for use order alone.

7 Conclusion

  • Across 87 SkillsBench tasks, the study combines controlled execution, Skill-content comparisons, and trajectory analysis to examine downstream utility.It finds that utility varies across configurations, relevance rankings do not fully reflect candidate utility, and operation-support reranking improves first-choice pass rates.Source: §7 · p. 12
  • With the Skill set unchanged, schemes specifying stage objectives or artifact dependencies achieve higher overall pass rates than schemes specifying order alone.The study presents case evidence for authoring practices on deliverable verification, revalidation after changes, and dependency recovery within a five-stage framework.Source: §7 · p. 12
  • The findings inform decisions about Skill authoring, selection, and organization by linking content and actual use to task outcomes.Source: §7 · p. 12
Loading 2610.08875v2…