PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks

Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Shenghan Zheng, Kyoung Whan Choe, Jiankai Sun, Shuyi Wang, Chujun Tao, Binxu Li, Xuandong Zhao, Hejia Geng, Xiaojun Wu, Junwei Zhou, Xiaokun Chen, Hanwen Xing, Yubo Li, Qunhong Zeng, Di Wang, Yuanli Wang, Roey Ben Chaim, Penghao Jiang, Haotian Shen, Luyang Kong, Xinyi Liu, Runhui Wang, Xuanqing Liu, Jiachen Li, Xin Lan, Yueqian Lin, Wengao Ye, Junwei He, Songlin Li, Yue Zhang, Yipeng Gao, Yijiang Li, Ze Ma, Liqiang Jing, Tianyu Wang, Kaixin Li, Yiqi Xue, Haoran Lyu, Yizhuo He, Yuchen Tian, Shutong Wu, Bowei Wang, Yixuan Gao, Bo Chen, Litong Liu, Sikai Cheng, Jiajun Bao, Shuaicheng Tong, Shuwen Xu, Terry Yue Zhuo, Tinghan Ye, Qi Qi, Miao Li, Longtai Liao, Zelin Tan, Chang Shi, Xilin Tang, Srinath Tankasala, Boqin Yuan, Yaoyao Qian, Jianhong Tu, Chenguang Wang, Yizhou Sun, Wei Wang, Aaron Taylor, Ziyue Yang, Changkun Guan, Zhikang Dong, Xinyu Zhang, Steven Dillmann, Han-chung Lee, Dawn Song

arXiv:2602.12670v4cs.AI

TL;DR

SkillsBench addresses the lack of systematic evidence about when Agent Skills help agents. It introduces a paired benchmark with curated Skills and deterministic verification, finding that Skills improve performance substantially but unevenly across configurations and tasks.

  • Problem

    Existing benchmarks do not systematically measure how much Skills help or when they fail, leaving the deployment question unanswered.

  • Method

    SkillsBench evaluates 87 containerized tasks across 8 domains under matched no-Skills and Skills-augmented conditions, using curated contributor-sourced Skills and deterministic verifiers.

  • Results

    Curated Skills raise task-macro pass rate from 33.9% to 50.5% (+16.6 pp) across the 87-task, 18-configuration aggregate, with configuration-level gains ranging from +4.1 pp to +25.7 pp.

  • Takeaways & Limitations

    Skills are most helpful for specialized, procedure-dependent workflows, while concise procedural Skills can partially substitute for model scale and exhaustive documentation can underperform.

  • Takeaways & Limitations

    Results may not transfer directly beyond terminal-based, containerized tasks to GUI agents, multi-agent coordination, or very long-horizon workflows.

Abstract

from arXiv · show

Agent Skills are structured packages of procedural knowledge that augment large language model (LLM) agents at inference time. Despite rapid adoption, there is no standard way to measure whether they actually help. We present SkillsBench, a benchmark whose current inventory contains 87 tasks across 8 domains paired with curated Skills and deterministic verifiers. Our latest aggregate evaluation runs the 87-task benchmark under matched no-Skills and curated-Skills conditions for 18 model-harness configurations. Curated Skills raise the average pass rate from 33.9% to 50.5% (+16.6 percentage points; 25.5% normalized gain), with configuration-level gains ranging from +4.1 to +25.7 pp. Focused Skills with at most three modules outperform larger or exhaustive bundles, and smaller models with Skills can match larger models without them. SkillsBench establishes paired evaluation as the foundation for rigorous measurement of Skill efficacy on agentic, expertise-heavy work.

1 Introduction

Agent Skills package procedural expertise for inference-time use, addressing the lack of systematic evidence about whether and when they improve agent performance. SKILLSBENCH introduces a paired benchmark to measure that effect directly.

  • Motivation: Agent Skills augment agents at inference time with structured instructions, code templates, resources, and reference material without modifying model weights.They encode standard operating procedures, domain conventions, and task heuristics as modular artifacts mediated by the agent harness.
  • Research gap: Existing benchmarks measure raw capability but do not isolate how much a Skill helps on a task.They fold model, harness, and augmentation effects into a single pass rate, leaving the deployment question unanswered.
  • Contribution: SKILLSBENCH treats Skills as first-class evaluation artifacts and measures them under matched no-Skills and curated-Skills conditions.The benchmark is designed to quantify augmentation effects rather than raw task performance alone.
  • Contribution: Curated Skills lift task-macro pass rate from 33.9% to 50.5% across 87 tasks and 18 model–harness configurations.The average improvement is +16.6 pp, with configuration-level gains ranging from +4.1 to +25.7 pp.

2 Background

The paper frames Skills as reusable, file-based procedural augmentation that becomes actionable through an agent harness. Compared with other runtime augmentation paradigms, Skills combine modularity, procedural guidance, executable resources, and portability.

  • Definitions: A Skill is a reusable, file-system-based procedural package for a class of agent tasks.It requires a SKILL.md file and may include scripts, templates, reference files, or worked examples.
  • Definitions: Skill augmentation makes relevant Skills available to an agent at inference time through its harness.The approach provides procedural guidance, can include executable resources, and represents Skills as portable files rather than model parameters.
  • Definitions: An agent harness wraps an LLM, connects it to the environment, manages tools and workspace state, and discovers and loads Skills.The harness is the execution layer through which file-based Skills become usable during inference.
  • Comparison: Skills combine modular packaging, procedural guidance, executable resources, and portability among runtime augmentation paradigms.This comparison distinguishes Skills from augmentation that supplies factual context alone.
  • Ecosystem: The construction snapshot contains 2,014,000 source-partitioned Skills, illustrating the scale of the surrounding ecosystem.Figure 2 presents this ecosystem count alongside the benchmark construction and evaluation pipeline.

3 SKILLSBENCH

SKILLSBENCH builds a filtered, reproducible benchmark of authentic tasks whose Skills provide general expertise rather than instance-specific answers. Each task pairs a containerized environment and deterministic verifier with controlled Skill-access conditions across 87 tasks and 8 domains.

  • Construction: SKILLSBENCH retains 87 tasks across 8 domains from 400 submissions by 142 contributors after automated gates and human review.Each task is containerized and evaluated under matched no-Skills and Skills-augmented conditions.
  • Skill boundary: Contributors author Skills independently of the benchmark, and task instructions do not name which Skills agents should use.These constraints keep Skills focused on pre-existing expertise rather than hidden answer keys.
  • Task principles: Tasks represent authentic real work and are evaluated with deterministic pass/fail tests rather than an LLM judge.The benchmark rejects classroom-style tasks, made-up scenarios, toy datasets, and purely synthetic data.
  • Task specification: Each task includes a human-authored instruction, Docker environment, oracle reference solution, and deterministic pytest verifier.Skill folders use a required SKILL.md plus optional scripts, references, and assets.
  • Composition: The inventory contains 6 Core, 53 Extended, and 28 Extreme tasks stratified by estimated human-specialist completion time.Per-domain counts range from N = 5 to N = 16.
  • Composition: Figure 4 organizes the 87 tasks across 8 domains and shows their difficulty stratification, with greater weight on production workflows than classroom problem sets.Domain definitions and per-task mapping appear in Table 19.

4 Experiment

The experiment compares matched agent configurations with no Skills and curated Skills on the same reproducible tasks. BenchFlow standardizes containers, Skill injection, sandboxing, trajectory logging, and deterministic verification across 18 configurations.

  • Configurations: The latest aggregate evaluates 18 configurations across OpenHands, Gemini CLI, Claude Code, and Codex CLI.Each task is run under no-Skills and curated-Skills conditions.
  • Protocol: BenchFlow creates a fresh pinned container for each configuration, task, and condition before running the agent to final submission.The deterministic test-script verifier then emits a pass/fail result.
  • Metrics: Task-macro pass rate averages verifier-scored outcomes across repeated trials within each task and then across the fixed 87-task inventory.Normalized gain complements absolute pass-rate deltas.
  • Metrics: Normalized gain measures the fraction of remaining performance headroom that Skills close.For multi-configuration summaries, normalized gain is computed per configuration before macro-averaging.

5 Results

Across the 87-task benchmark, curated Skills improve performance broadly but unevenly across configurations, harnesses, domains, and tasks. Results also show that focused Skills generally outperform exhaustive bundles, while self-generated Skills can hurt performance and high Skill access does not guarantee task resolution.

  • Every one of the 18 model–harness configurations improves with curated Skills, with gains ranging from +4.1 to +25.7 pp.The average gain is +16.6 pp, but efficacy depends on the specific agent stack.
  • The strongest with-Skills pass rates are 67.3% for OpenHands + GPT-5.5, 66.5% for Codex + GPT-5.5, and 61.2% for Claude Code + Opus 4.7.
  • Harness choice materially changes Skill outcomes: Gemini 3.1 Pro reaches 60.8% in Gemini CLI but 52.8% in OpenHands, while Claude Opus 4.7 reaches 61.2% versus 53.1%.The harness mediates discovery, prompting, execution, and tool use.
  • High Skill Invocation Rate does not guarantee high resolution, indicating that failures often occur after agents access task-specific Skills.Figure 5 separates task-specific Skills from harness-bundled Skills and measures invocation alongside resolution.
  • Curated Skills outperform self-generated Skills on all three dedicated-harness configurations, while self-generated Skills fall 8.1–11.5 pp below their no-Skills baselines.Audited causes include undiscovered generated packs, displaced solver work, and confidently wrong pack content.
  • Domain gains are largest in Natural Science (+28.8 pp), Media & Content Production (+24.1 pp), and Cybersecurity (+18.9 pp), and smallest in Software Engineering (+11.6 pp) and Mathematics & OR (+9.7 pp).The benchmark reports positive aggregate deltas across domains.
  • The 10 highest-improvement tasks average +67.0 pp, including llm-prefix-cache-replay at 94.4% from 1.9% and dapt-intrusion-detection at 81.5% from zero.
  • 13 of 87 tasks show negative Skills deltas, with the largest drops reaching −7.4 pp.Audits identify heavyweight pipelines, displaced stronger defaults, and solvers the agent cannot debug as repeatable patterns.

6 Discussion

The discussion finds that Skills help most when tasks require concrete procedural and verifier-facing details, but efficacy depends on harnesses and documentation design. It also identifies complexity, transfer, attribution, and ecological-validity boundaries for the benchmark’s conclusions.

  • Skills help most when success depends on concrete procedures and verifier-facing details rather than broad conceptual knowledge.Benefits are largest for specialized workflows or brittle formats.
  • Harness implementation mediates Skill efficacy because some harnesses retrieve and invoke Skills reliably while others do not.Long trajectories can also suffer format drift that weakens early-injected guidance.
  • Implications for Skill authoring: Focused documentation with verifier-facing detail yields larger aggregate lift than comprehensive prose.The reported aggregate lifts are +0.7 pp for comprehensive prose versus +19.0/+21.5 pp for focused documentation.
  • Complexity-aware fallback paths for Skills: Negative Skill deltas share a root cause: pipelines that are correct in principle but too heavy or brittle for routine agent execution.The proposed complexity contract would state costs, applicability boundaries, and lightweight fallback paths.
  • Coverage and generalization: Results may not transfer directly beyond terminal-based, containerized tasks or the limited models and harnesses evaluated.The paper identifies GUI agents, multi-agent coordination, and very long-horizon workflows as important extension settings.
  • Causal attribution and controls: Observed gains may partly reflect increased context rather than procedural structure, while containerization cannot eliminate nondeterminism or training-set leakage.The paper calls for length-matched controls and ecosystem-representative Skills to isolate causal components and improve ecological validity.

7 Related Work

SKILLSBENCH builds on agent benchmarks, procedural augmentation, Skills ecosystems, and paired evaluation. Its methodological distinction is applying matched with-versus-without comparisons to independently authored Skills across multiple expertise-heavy domains.

  • Agent benchmarks: Existing agent benchmarks evaluate end-to-end capability across realistic environments, including terminal, software-engineering, web, GUI, and tool-mediated workflows.
  • Procedural augmentation and tool use: Prior procedural-augmentation work supplies structured reasoning, external knowledge, retrieval, tool use, and declarative optimization frameworks.
  • Skills ecosystems and evaluation methodology: Agent Skills and MCP specifications formalized Skill packages and tool connectivity, while agent CLIs provide real-world harnesses.SKILLSBENCH runs on BenchFlow and uses Terminal-Bench scoring for comparability.
  • Paired evaluation of agent augmentations: SKILLSBENCH adopts paired with-versus-without evaluation while decoupling tasks from independently authored Skills and spanning eight expertise-heavy domains.This extends paired evaluation beyond a single enterprise API workflow class.

8 Conclusion

The conclusion presents SKILLSBENCH as a paired benchmark showing substantial but uneven benefits from curated Skills, with concise procedural Skills outperforming exhaustive documentation. It also reports the scale and provenance of the construction corpus and the timestamped temporal snapshot used for analysis.

  • 8 Conclusion: Curated Skills improve performance by +16.6 pp on average across the 87-task, 18-configuration aggregate, with gains ranging from +4.1 pp to +25.7 pp.The comparison is between no-Skills and curated-Skills conditions.
  • 8 Conclusion: Concise procedural Skills outperform exhaustive documentation, and Skills can partially substitute for model scale on procedural tasks.
  • 8 Conclusion: The construction snapshot contains 2,014,000 source-partitioned Skills aggregated from direct source discovery, marketplace records, and partner inventories.Entries were deduplicated within source partitions before summing.
  • 8 Conclusion: Per-Skill structural statistics use 767,430 cloned bundles whose source repositories remained reachable.Other entries lacked cloneable or reachable source for reasons including deletion, privacy, or fetch failure.
  • 8 Conclusion: Daily Skill additions in the timestamped snapshot peaked at 33,692 on 2026-04-07.The timestamped window begins on 2025-09-03 and reaches its first day with at least 100 additions on 2025-10-16.

A.2 Skill Characteristics

SkillsBench characterizes Skills as generally concise, documentation-heavy procedural artifacts drawn from a broad ecosystem, while its benchmark uses high-quality, curated Skills and strict task packaging.

  • Domain coverage: 22.4% of marketplace assignments are Tools, 17.0% Business, and 14.3% Development, with no category exceeding a quarter of assignments.The remaining assignments span testing and security, data and AI, DevOps, documentation, content and media, and a long tail.
  • Structure: 59.9% of Skills contain a single file, and 86.4% contain five or fewer files, indicating that most bundles are minimal.The median bundle contains 1.8k tokens, while the size distribution has a dramatic long tail.
  • Structure: 51.7% of sampled files are Markdown, followed by Python, TypeScript, JavaScript, and JSON, showing that natural-language instructions dominate executable implementation.The sample contains 4.27M files across 1,825 extensions.
  • Quality: The ecosystem’s mean quality score is 6.2/12, based on completeness, clarity, specificity, and examples.This score indicates substantial variation in authoring quality.
  • Benchmark design: Benchmark Skills were selected from the top quality quartile, with scores of at least 9/12, and their median size informed an 8K-token context budget.The selection was intended to isolate procedural knowledge from Skill-quality variance.
  • Benchmark design: The 87-task benchmark is an optimistic scenario because its Skills average 10.1/12, versus 6.2/12 across the ecosystem, and matching is imperfect.The authors call for ecosystem-representative Skill samples in future evaluations.

D.10 Experiment Orchestration

The experiment orchestrates a fixed, paired evaluation of 87 tasks across model configurations, with task-level visualizations and ablations examining bundle size, documentation length, and negative deltas.

  • Evaluation frame: 9,396 fixed result slots cover 18 configurations, 87 tasks, and two matched Skills conditions.The released evaluation uses three selected public result files per task and condition, with 256 concurrent trials for the main conditions.
  • Task-level orchestration: Figures 11–13 show task-level pass rates using a fixed three-trial denominator, with tasks and models sorted by their with-Skills performance.The figures compare curated Skills, no Skills, and per-task uplift.
  • Task-level orchestration: The majority of Figure 13 cells show positive Skill uplift, although a small number of tasks have negative deltas for specific models.Figure 11 also shows easy tasks solved broadly and hard tasks remaining unsolved as model capability decreases.
  • Bundle ablations: One to three attached Skills remain strongest, while four or more introduce diminishing returns consistent with cognitive overhead and conflicting guidance.The benchmark therefore favors focused bundles over exhaustive collections.
  • Documentation ablations: Focused documentation outperforms comprehensive prose when Skills are grouped by SKILL.md length.The buckets use the 25th, 50th, and 95th percentile cutpoints.
  • Failure analysis: Negative-delta trajectory audits identify heavyweight pipelines that crowd out simpler execution as a recurring failure pattern.The same audit covers tasks where Skill activation displaces stronger native strategies and extracts authoring patterns from the ten highest-improvement tasks.

I Comprehensive Results Summary

Across the complete aggregate, curated Skills improve performance for every evaluated configuration, with gains appearing broadly across task families and especially on expertise-heavy tasks.

  • Aggregate results: +16.6 pp is the average curated-Skills improvement across 18 configurations, with gains ranging from +4.1 to +25.7 pp.Scores use the fixed 87 × 3 trial frame per condition.
  • Configuration results: OpenHands + GPT-5.5 reaches the highest absolute pass rate, at 67.3% with Skills.OpenHands + GLM 5.1 has the largest absolute improvement, while Gemini CLI + Gemini 3.1 Pro has the highest normalized gain.
  • Aggregate results: Every model family improves with curated Skills, with no material time cost.The family-level comparison is shown in the time–performance plane.
  • Task examples: SEC 13F reporting rises from 0% without Skills to 68.5% with Skills (+68.5 pp) after the Skill documents regulatory formats and comparison procedures.The guidance covers SEC EDGAR endpoints, 13F-HR structure, and cross-quarter comparisons.
  • Task examples: Manufacturing-fjsp-optimization moves from a 0% baseline toward successful constraint-aware solving when the Skill specifies propagation, objectives, and OR-Tools configuration.Without Skills, agents produced schedules that ignored maintenance constraints.
  • Task examples: Skills bridge domain-specific API gaps, raising sales-pivot-analysis from 1.9% to 40.7% (+38.9 pp).The curated Skill supplied Office-specific guidance for programmatic pivot-table creation.

J.3 Tasks With Largest Skills Impact

The largest Skill effects occur on tasks requiring domain-specific procedures, while the broader evaluation also measures invocation, resource use, runtime errors, and artifact structure.

  • Largest impacts: +67.0 percentage points is the average improvement across the ten tasks with the largest Skills impact.These tasks involve cache behavior, intrusion detection, financial schemas, scientific pipelines, optimization, and artifact conversion.
  • Evaluation diagnostics: The benchmark reports Skill invocation rates, token usage, costs, and trial outcomes across the latest 18-configuration aggregate.The tables distinguish task-bundled Skills from harness-bundled Skills and use selected public trials.
  • Evaluation diagnostics: 78 of 9,396 trials, or 0.8%, are runtime errors with no scored result.Runtime errors are classified from recorded error-category fields.
  • Resource use: Skills change token usage in both directions: some configurations consume more tokens, while others consume fewer.Examples include Gemini 3.1 Pro increasing from 1.82M to 4.02M tokens and Claude Opus 4.7 decreasing from 1.77M to 1.09M.
  • Time–performance trade-off: Curated Skills raise fleet resolution by +16.6 pp at broadly unchanged agent time, while OpenHands + Claude Opus 4.7 falls from 10.0 to 6.5 minutes per task.The 4–9-minute band contains both the strongest and weakest configurations.

M.9 Domain Taxonomy and Per-Task Mapping

SkillsBench organizes its 87 released tasks into eight broad professional domains while retaining finer capability and difficulty metadata. The inventory mapping is generated automatically to remain synchronized with the benchmark’s figures.

  • Domain taxonomy: 87 tasks are classified into eight domains using broad professional-context labels and finer task metadata.Cybersecurity and Media & Content Production currently have fewer than eight tasks each.
  • Scope boundary: Cybersecurity (n = 7) and Media & Content Production (n = 5) are below N = 8, so domain-level inferential claims for these slices should remain descriptive.Additional tasks are needed before stronger per-domain inference is warranted.
  • Task metadata: Each task receives a primary capability axis—Reasoning, Agentic Coding, Multimodal, Tool Use, or Search & Research—and a difficulty marker.Difficulty markers distinguish Core, Extended, and Extreme tasks by expected duration.
  • Domain coverage: The taxonomy spans professional domains including Office & White Collar, Finance & Economics, Mathematics & OR, Cybersecurity, and Media & Content Production.Examples include document and spreadsheet work, financial analysis, formal proofs, security remediation, and video or 3D production.
  • Representative tasks: The released inventory includes tasks such as citation checking, PDF editing, spreadsheet recovery, proof verification, routing, vulnerability fixing, and video dubbing.These examples illustrate the breadth of capability axes and professional contexts represented in the mapping.
Loading 2602.12670v4…