PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks

Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Shenghan Zheng, Kyoung Whan Choe, Jiankai Sun, Shuyi Wang, Chujun Tao, Binxu Li, Xuandong Zhao, Hejia Geng, Xiaojun Wu, Junwei Zhou, Xiaokun Chen, Hanwen Xing, Yubo Li, Qunhong Zeng, Di Wang, Yuanli Wang, Roey Ben Chaim, Penghao Jiang, Haotian Shen, Luyang Kong, Xinyi Liu, Runhui Wang, Xuanqing Liu, Jiachen Li, Xin Lan, Yueqian Lin, Wengao Ye, Junwei He, Songlin Li, Yue Zhang, Yipeng Gao, Yijiang Li, Ze Ma, Liqiang Jing, Tianyu Wang, Kaixin Li, Yiqi Xue, Haoran Lyu, Yizhuo He, Yuchen Tian, Shutong Wu, Bowei Wang, Yixuan Gao, Bo Chen, Litong Liu, Sikai Cheng, Jiajun Bao, Shuaicheng Tong, Shuwen Xu, Terry Yue Zhuo, Tinghan Ye, Qi Qi, Miao Li, Longtai Liao, Zelin Tan, Chang Shi, Xilin Tang, Srinath Tankasala, Boqin Yuan, Yaoyao Qian, Jianhong Tu, Chenguang Wang, Yizhou Sun, Wei Wang, Aaron Taylor, Ziyue Yang, Changkun Guan, Zhikang Dong, Xinyu Zhang, Steven Dillmann, Han-chung Lee, Dawn Song

arXiv:2602.12670v4cs.AI

TL;DR

Agent Skills have proliferated without a benchmark that systematically measures their task-specific benefit. SkillsBench compares matched no-Skills and Skills-augmented conditions on 87 tasks across eight domains using deterministic verifiers. Curated Skills improve performance substantially but unevenly across configurations, with an average gain of +16.6 pp.

  • Problem

    Despite proliferating Skills, no benchmark systematically measures whether adding a Skill helps an agent on a particular task and by how much.

    Source: §1 · p. 2

  • Method

    SkillsBench compares matched no-Skills and Skills-augmented conditions on 87 tasks across eight domains using deterministic verifiers to isolate Skills’ contributions.

    Source: §3 · p. 3

  • Results

    Across tasks and configurations, curated Skills improve performance substantially but unevenly (+16.6 pp on average).

    Source: §8 · p. 9

  • Takeaways & Limitations

    Concise procedural Skills outperform exhaustive documentation and can partially substitute for model scale on procedural tasks.

    Source: §8 · p. 9

    Observed gains may partly reflect increased context length rather than procedural structure, requiring stronger length-matched baselines for causal attribution.

    Source: §6.1 · p. 8

Abstract

from arXiv · show

Agent Skills are structured packages of procedural knowledge that augment large language model (LLM) agents at inference time. Despite rapid adoption, there is no standard way to measure whether they actually help. We present SkillsBench, a benchmark whose current inventory contains 87 tasks across 8 domains paired with curated Skills and deterministic verifiers. Our latest aggregate evaluation runs the 87-task benchmark under matched no-Skills and curated-Skills conditions for 18 model-harness configurations. Curated Skills raise the average pass rate from 33.9% to 50.5% (+16.6 percentage points; 25.5% normalized gain), with configuration-level gains ranging from +4.1 to +25.7 pp. Focused Skills with at most three modules outperform larger or exhaustive bundles, and smaller models with Skills can match larger models without them. SkillsBench establishes paired evaluation as the foundation for rigorous measurement of Skill efficacy on agentic, expertise-heavy work.

1 Introduction

SkillsBench evaluates Agent Skills through matched with- and without-Skills comparisons, finding an average improvement of +16.6 percentage points across 18 model–harness configurations. It also provides a paired-evaluation framework generalizable to other agent augmentations.

1 Introduction

  • Foundation models lack specialist procedural knowledge, while domain-specific fine-tuning is expensive and sacrifices generality [Brown et al., 2020, Ouyang et al., 2022, Yao et al., 2023b].This tension motivates augmentation for agents deployed in production and expertise-heavy workflows.Source: §1 · p. 2
  • Agent Skills package instructions, code templates, resources, and reference material to augment agents at inference time without changing model weights.They encode operating procedures, domain conventions, and task heuristics as modular artifacts mediated by the agent harness [Sutton et al., 1999, Sumers et al., 2024].Source: §1 · p. 2
  • 2,014,000 source-partitioned Skills populate the construction snapshot, yet no benchmark systematically measures how much Skills help or when they fail.The missing deployment-level question is whether adding a particular Skill helps a particular agent on a task, and by how much.Source: §1 · p. 2
  • 33.9% to 50.5% task-macro pass rate (+16.6 pp; 25.5% normalized gain) is the curated-Skills improvement over no Skills across 18 model–harness configurations.Matched evaluation covers 87 tasks, with configuration-level improvements ranging from +4.1 to +25.7 pp.Source: §1 · p. 2
  • SkillsBench treats Skills as first-class evaluation artifacts and provides paired evaluation generalizable to retrieval pipelines, memory stores, and scaffolding.Contributor-driven sourcing, leakage-controlled task-to-artifact decoupling, and the BenchFlow [BenchFlow team, 2026] containerized harness avoid confounding model and augmentation effects. The benchmark, harness, and public trajectories/results are open-sourced.Source: §1 · p. 2

2 Background

Skills are reusable, file-system-based procedural packages that augment agents at inference time. A compatible agent harness makes their instructions and resources actionable.

Skill Definition

  • A Skill is a reusable procedural package for a class of agent tasks, containing a required SKILL.md file with natural-language instructions.Optional auxiliary resources include scripts, templates, reference files, and worked examples.Source: §2 · p. 2

Skill Augmentation

  • Skill augmentation provides modular, reusable procedural guidance at inference time, can include executable resources, and supports cross-model portability through file-based representation rather than model parameters.It supplies procedural guidance rather than factual context alone.Source: §2 · p. 2, Table 1 · p. 3

Agent Harness

  • An agent harness connects an LLM to its environment [Lopopolo, 2026, Lee, 2026], managing tools, workspace state, script execution, observations, and Skill loading.A Skill is neither model weights nor an executable tool by itself; a compatible harness injects its instructions and exposes its resources.Source: §2 · p. 2

3 SKILLSBENCH

SkillsBench measures Skill efficacy through matched evaluations on containerized, deterministically verified tasks representing authentic work. Its construction pipeline checks instruction provenance and screens Skills for task-specific answer leakage.

Benchmark scope

  • 87 tasks across 8 domains, selected from 400 submissions by 142 contributors, form the evaluated inventory for matched no-Skills and Skills-augmented comparisons.Estimated unaided specialist completion times divide the inventory into 6 Core tasks (<60 min), 53 Extended tasks (1–4 h), and 28 Extreme tasks (>4 h).Source: §3 · pp. 3–4

Skills as expertise, not answers

  • Skills must provide reusable domain expertise rather than instance-specific answers, be authored independently of the benchmark, and be discovered without task instructions naming them.Agents discover and activate Skills through the standard progressive-disclosure mechanism.Source: §3 · p. 3

Task principles

  • Tasks must represent authentic work, use human-authored instructions and deterministic pass/fail verification, and admit end-to-end oracle solutions without pre-baked answers or hard-coded numbers.Difficulty must arise from the problem rather than confusing instructions; classroom exercises, invented scenarios, toy datasets, and purely synthetic data are rejected.Source: §3 · p. 3

Evaluation design

  • Three evaluation arms vary only Skill access: no Skills, a discoverable curated bundle, or agent-authored Skill documents used during solving.The same deterministic pytest battery scores each arm by its fraction of checks passed, rA, rB, and rC. Curated and self-authored Skill efficacy are measured against the same baseline: ∆curated = rB − rA and ∆self = rC − rA.Source: §3 · p. 3, Figure 3 · p. 4

Sourcing and filtering

  • Four automated gates screen submissions for structural integrity, oracle execution, instruction provenance, and Skill-to-solution leakage before human review.Provenance screening combines an AI-text detector with human labeling, yielding a 100% human-authored release set. Passing submissions receive at least one 30-minute maintainer review; tasks without measurable separation between conditions are rejected as low-signal.Source: §3 · p. 4

Task specification

  • Each task packages a human-authored instruction, Docker environment with task data and Skills, oracle reference solution, and deterministic pytest verifier.Tasks follow the Agent Skills format; each Skill folder requires SKILL.md and may include scripts, references, and assets. The oracle reference solution must pass the verifier.Source: §3 · p. 4

4 Experiment

The experiment compares matched no-Skills and curated-Skills conditions across 18 configurations using deterministic verification on a fixed 87-task inventory. Task-macro pass rate measures performance, while normalized gain measures the remaining headroom Skills close.

Models, harnesses, and conditions

  • 18 configurations across four terminal-agent harnesses compare matched no-Skills and curated-Skills conditions.Harnesses are OpenHands, Gemini CLI, Claude Code, and Codex CLI; conditions provide task instructions alone or the full environment/skills/ directory. Separately, three dedicated-harness configurations evaluate self-generated Skills: agents author packs with Anthropic’s skill-creator, then solve using only those packs.Source: §4 · p. 5

Evaluation harness and protocol

  • BenchFlow [BenchFlow team, 2026] provides uniform Skill injection, sandboxing, trajectory logging, and deterministic pass/fail verification.Each configuration–task–condition run uses a fresh pinned container and continues until final submission. Unscored, stale, or rate-limited trajectories are incomplete coverage for rerun/audit; timeout rows are scored as failures only when healthy pass/fail replacements are unavailable.Source: §4 · p. 5

Metrics

  • Task-macro pass rate averages trial outcomes within each task, then across the fixed 87-task inventory, following Terminal-Bench [Merrill et al., 2026].The latest aggregate targets three selected public trials per configuration–task–condition cell. Conditions use the same task and container, making deltas paired differences at the configuration–task level rather than unpaired-pool differences.Source: §4 · p. 5
  • Normalized gain measures the fraction of remaining performance headroom closed by Skills.For multi-configuration summaries, normalized gains are computed separately at full precision and macro-averaged. Consequently, reported mean gain can differ slightly from gain recomputed using mean pass rates.Source: §4 · p. 5

5 Results

Curated Skills improve every evaluated model–harness configuration, but benefits vary across harnesses, domains, tasks, and Skill designs. Focused Skills outperform exhaustive guidance, while self-generated Skills underperform no-Skills baselines.

5.1.1 Main Results

  • +16.6 pp is the average curated-Skills gain across 18 model–harness configurations; every configuration improves, with gains ranging from +4.1 to +25.7 pp.OpenHands + GPT-5.5 achieves the highest with-Skills pass rate (67.3%), whereas OpenHands + GLM 5.1 achieves the largest lift (+25.7 pp; 38.1% normalized gain). Strong baseline performance does not guarantee large gains: OpenHands + Gemini 3.5 Flash starts at 41.1% but gains only +7.1 pp.Source: §5.1.1 · p. 5, Table 2 · p. 6
  • 60.8% with-Skills performance for Gemini 3.1 Pro in Gemini CLI versus 52.8% in OpenHands demonstrates that harness choice materially changes Skill benefits.Claude Opus 4.7 similarly reaches 61.2% in Claude Code versus 53.1% in OpenHands. Harness discovery, prompting, execution, and tool-use loops mediate realized benefit.Source: §5.1.1 · p. 6

5.1.2 Domain-Level Analysis

  • +28.8 pp in Natural Science leads domain gains, compared with +9.7 pp in Mathematics & OR; all domains show positive aggregate gains.Media & Content Production gains +24.1 pp, Cybersecurity +18.9 pp, and Software Engineering +11.6 pp. Domains requiring specialized procedural knowledge underrepresented in pretraining benefit most, whereas stronger pretraining and tooling coverage correspond to smaller benefits.Source: Table 3 · p. 7, §5.1.2 · p. 7

5.1.1 Main Results

  • High task-specific Skill invocation does not guarantee resolution, suggesting discovery is usually not the bottleneck and failures often persist after Skill access.Invocation counts cover reads or invocations of task-specific Skills under each task’s environment/skills, excluding harness-bundled Skills.Source: §5.1.1 · pp. 6–7, Figure 5 · p. 6
  • −8.1, −11.3, and −11.5 pp are self-generated-Skills deficits versus no-Skills for Claude Code + Opus 4.7, Codex + GPT-5.5, and Gemini CLI + Gemini 3.1 Pro, respectively.Curated Skills instead add +18.2 to +24.8 pp on these same configurations. Audits attribute self-generated deficits to undiscovered packs, authoring that displaces solver work, and confidently incorrect pack content.Source: §5.1.1 · p. 7

5.1.3 Task-Level Analysis

  • 13 of 87 tasks regress with Skills, while the 10 largest task-level beneficiaries average +67.0 pp improvement.Audits identify heavyweight pipelines, displacement of stronger defaults, and solvers agents cannot debug, reflecting prescribed pipelines without applicability boundaries or lightweight fallbacks. Conversely, llm-prefix-cache-replay reaches 94.4% from a 1.9% no-Skills pass rate.Source: §5.1.3 · p. 7
  • +19.0 pp for tasks with 2–3 Skills exceeds +10.1 pp for ≥4 Skills, while focused procedural guidance outperforms exhaustive documentation.Tasks with one Skill gain +18.0 pp. Compact and standard-length Skills gain +19.0 and +21.5 pp, respectively, versus +14.5 pp for detailed Skills and +0.7 pp for comprehensive documentation.Source: §5.1.3 · pp. 7–8
  • 34.9% for MiniMax M2.7 with Skills exceeds no-Skills pass rates for OpenHands + GLM 5.1 (32.7%) and OpenHands + MiniMax M3 (29.7%).Skills can therefore partly compensate for model capacity on procedural tasks.Source: §5.1.3 · p. 8

6 Discussion

Skills primarily close procedural gaps, with effectiveness shaped by harness implementation and focused, verifier-facing guidance. Generalization and causal attribution remain limited, motivating broader evaluation and stronger controls.

6 Discussion

  • Skills help most with concrete procedures and verifier-facing details, especially in specialized workflows or brittle formats, rather than broad conceptual knowledge.Gains are smaller or negative when models already have strong priors or Skills introduce overhead.Source: §6 · p. 8
  • Harness implementation mediates Skills efficacy through retrieval, actual invocation, and long-trajectory failures, making evaluation across multiple harnesses necessary.Some harnesses acknowledge Skills without invoking them; structured-interface format drift can weaken the influence of early-injected Skills.Source: §6 · p. 8
  • +0.7 pp aggregate lift from comprehensive prose contrasts with +19.0/+21.5 pp from focused documentation, favoring verifier-facing detail over completeness.Auditing the 10 highest-∆ tasks identified five recurring patterns: executable scripts with calibrated defaults, canonical sources and parsing quirks, verifier-facing file-format constraints, algorithmic invariants, and task-specific description: frontmatter for first-scan matching.Source: §6 · p. 8
  • The authors propose a SKILL.md complexity contract specifying expected tool and token costs, applicability boundaries, and a required lightweight fallback.The three failure modes driving negative deltas share pipelines that are correct in principle but too heavy or brittle for routine execution. Explicit contracts would help agents bypass unjustifiably expensive workflows.Source: §6 · p. 8

6.1 Limitations and Future Work

  • Terminal-based, containerized tasks and a limited model–harness sample constrain generalization to GUI agents, multi-agent coordination, and very long-horizon workflows.Skills integration can change over time. Proposed extensions include multimodal Skills and protocols for vision-language agents in GUI environments.Source: §6.1 · p. 8
  • Longer injected context complicates causal attribution, requiring length-matched controls to distinguish procedural structure from additional context.Self-generated Skills do not reproduce curated-Skills gains, but this comparison confounds content quality, discovery, and creator/solver interference. Proposed controls include random or irrelevant text and retrieval-only documentation to isolate component effects and study automatic synthesis.Source: §6.1 · p. 8
  • Containerization does not eliminate nondeterminism or training-set leakage, while broader ecological validity and interactions among composed Skills remain open questions.Multiple runs, a leakage audit, and paired comparisons mitigate but cannot eliminate these risks. Future evaluations should include lower-quality and automatically selected Skills, testing when combinations help or interfere and whether atomic effects predict composite performance.Source: §6.1 · p. 8

7 Related Work

SkillsBench measures Skill augmentation efficacy through paired evaluation rather than fixed-agent task completion, situating this approach within procedural augmentation, Skills infrastructure, and evaluation methodology.

Agent benchmarks

  • Unlike Terminal-Bench [Merrill et al., 2026] and other fixed-agent completion benchmarks, SkillsBench measures augmentation efficacy through paired evaluation.The distinction is between measuring an agent’s task-completion capability and measuring whether an augmentation improves it.Source: §7 · p. 9

Procedural augmentation and tool use

Skills ecosystems and evaluation methodology

  • SkillsBench runs on BenchFlow [BenchFlow team, 2026] and uses Terminal-Bench scoring [Merrill et al., 2026] for comparability.Anthropic’s Agent Skills and MCP specifications [Anthropic, 2025a, 2024] formalized Skill packages and tool connectivity. Agent CLIs provide real-world harnesses [Anthropic, 2025b, Google, 2025, OpenAI, 2025].Source: §7 · p. 9

Paired evaluation of agent augmentations

  • Building on paired augmentation evaluation [Liu et al., 2023], SkillsBench separates tasks from independently authored Skills and broadens domain coverage.The precedent compares with-versus-without conditions on identical enterprise API tasks and reports paired bootstrap CIs; SkillsBench spans 8 expertise-heavy domains. SkillsBench reports absolute gains and normalized gain across No-Skills baselines.Source: §7 · p. 9

8 Conclusion · A Skill Ecosystem Analysis · A.1 Data Collection

Curated Skills substantially but unevenly improve SkillsBench performance, with concise procedural Skills outperforming exhaustive documentation. The ecosystem analysis describes a 2,014,000-Skill source-partitioned snapshot, with structural statistics collected from 767,430 reachable bundles.

8 Conclusion

  • +16.6 pp is the average curated-Skills improvement over no-Skills across 87 tasks and 18 configurations, with configuration gains ranging from +4.1 pp to +25.7 pp.SkillsBench evaluates Agent Skills as first-class artifacts using paired comparisons.Source: §8 · p. 9
  • Concise procedural Skills outperform exhaustive documentation and can partially substitute for model scale on procedural tasks.Source: §8 · p. 9
  • The authors release the SkillsBench benchmark, the BenchFlow harness, and public trajectories and results.Source: §8 · p. 9

A Skill Ecosystem Analysis

  • 2,014,000 source-partitioned Skills comprise 234,896 direct GitHub/source Skills, 1,773,213 marketplace Skills, and 5,891 partner Skills.Entries are deduplicated within each source before summing partitions. Analyses requiring repository timestamps or cloned files report their reachable sample sizes separately.Source: A Skill Ecosystem Analysis · p. 13

A.1 Data Collection

  • More than an order of magnitude of corpus growth followed the January 2026 snapshot of 37k Skills.Records include name, description, author, URL, source partition, and upstream repository pushed_at when available. These timestamps can reflect inherited repository history rather than actual Skill creation, making the pre-launch tail unreliable evidence of activity.Source: §A.1 · p. 13, Figure 6 · p. 13
  • 767,430 reachable Skill bundles were cloned to measure per-Skill size, file count, and extension mix.Other entries lacked cloneable source, referenced deleted or private repositories, or failed to fetch within retry limits.Source: §A.1 · p. 13

A.2 Skill Characteristics … D.9 Inference Configuration

These appendices characterize marketplace Skills and their quality and representativeness limitations, and document benchmark selection, task packaging, deterministic verification, review controls, and paired evaluation. They also compare self-generated and curated Skills across dedicated-harness configurations and audit distinct mechanisms behind self-generated Skills’ underperformance.

A.2 Skill Characteristics

  • 4.8 KB median SKILL.md instructions and 7.2 KB median bundles characterize heavy-tailed Skill sizes, with the largest bundle exceeding 1.6 GB.Single-file Skills account for 59.9% of bundles, 86.4% contain five or fewer files, and Markdown represents 51.7% of files. Across 12 marketplace categories, Tools accounts for 22.4% of assignments, Business 17.0%, and Development 14.3%, with no single dominant area.Source: §A.2 · pp. 13–14, Figure 7 · p. 14

A.3 Quality Indicators / A.4 Implications for Benchmark Design

  • 6.2/12 mean ecosystem quality contrasts with 10.1/12 for benchmark Skills, making the 87-task benchmark an optimistic scenario.The rubric scores completeness, clarity, specificity, and examples at 0–3 points each; ecosystem scores have SD=2.8, and real-world usage also involves imperfect Skill-task matching. Benchmark construction mirrors ecosystem domains, selects top-quartile Skills scoring ≥9/12, uses an 8K context allocation informed by median size, and incorporates quality-motivated leakage audits and authoring guidelines.Source: §A.4 · p. 15, §A.3 · p. 15

B.4 PR Review Process / B.6 Maintainer Review Policy / B.8 Benchmark Report Template / B.9 Review Lifecycle

  • 87 of 400 submissions from 142 contributors passed all review stages, a 22% acceptance rate.Maintainers assess data validity, professional realism, oracle and Skill quality, and anti-cheating robustness; policies prefer simple oracles, reject synthetic or toy data, and require personal inspection of multimodal outputs. Structured reports record oracle outcomes, paired agent performance and timing, failure evidence, and recommendations, while a label-based lifecycle tracks review, requested changes, re-review, and admission.Source: §B.4 · p. 17, §B.6 · p. 17, §B.8 · p. 18

B.5 Contributor Checklist / B.7 Automated CI Pipeline / B.10 Task Quality Criteria

  • Task admission requires realistic, Skill-dependent, deterministically verifiable work, reinforced by contributor requirements and automated quality checks.Contributors must provide human-written instructions checked with GPTZero and human review, correct and generalizable Skills, a fully passing oracle, paired agent tests, and fewer than 10 tests unless justified; tasks should exercise 3–6 Skills without naming them in instructions. CI validates structure, schema, container builds, and oracle execution, screens AI-generated instructions, and checks instruction–test consistency, anti-cheating measures, pinned dependencies, typos, and hardcoded solutions.Source: §B.5 · p. 17, §B.7 · p. 17, §B.10 · p. 18

B Task Specification and Review Process / B.1 Task Directory Structure / B.2 Task Configuration (task.md Frontmatter) / B.3 Verification Infrastructure / C The task.md Task Standard / C.1 Document Structure

  • The task.md standard combines configuration and instructions in one schema-validated document, supporting portable task packages and isolated creator–solver sessions.Extending Harbor, the format supports the self-generated-Skills condition, packages verifier/ and oracle/ directories, records metadata and resource limits in frontmatter, rejects unknown top-level keys, and loads all 87 tasks through one parser. Verification uses pytest with CTRF for 85 of 87 tasks and direct Python checkers for two; every test.sh writes a reward, with assertion-based verification passing only when all assertions succeed.Source: §B.2 · p. 16, §B.3 · p. 16, C The task.md Task Standard · pp. 18–19

C.2 Controlled Metadata Vocabulary / C.3 Explicit Network Policy

  • Explicit network policies constrain egress, while controlled metadata vocabularies are enforced during review rather than parsing.network_mode accepts no-network, public, or allowlist; allowlist requires a nonempty allowed_hosts list at parse time. Metadata combines controlled category, task_type, modality, interface, skill_type, and difficulty fields with open tags and Skill names, preserving authoring flexibility while supporting benchmark classification.Source: §C.3 · p. 19, §C.2 · p. 19

C.4 Verifier Strategies

  • 87 tasks exclusively use test-script verification, preserving deterministic measurement without LLM-as-a-judge variance despite the standard supporting additional verifier strategies.Verifier selection can use task.md or an optional verifier manifest; other supported strategies include llm-judge, reward-kit, and agent-judge / ors-episode, and task.md packages also run directly on AgentBeats. Unscored, stale, or rate-limited rows represent incomplete coverage; timeouts count as failures only when healthy replacements are unavailable.Source: C The task.md Task Standard · p. 18, §C.4 · pp. 19–20

D.1 Setup overview / D.2 Model and Harness Configurations / D.3 Harness Descriptions / D.4 Agent Interface

  • 9,396 selected public result files cover 18 configurations × 87 tasks × two conditions × three trials, with all runs using temperature 0.OpenHands, Gemini CLI, Claude Code, and Codex CLI use a standardized environment interface; following Terminal-Bench [Merrill et al., 2026], task-macro pass rate averages trials within tasks and then across tasks, alongside absolute deltas and normalized gain g = (passskill − passvanilla)/(1 − passvanilla). Claude models’ training awareness of the Agent Skills specification may advantage their processing of Skill-formatted instructions.Source: §D.1 · p. 20, §D.4 · p. 21, §D.3 · p. 21

D.4 Agent Interface / D.5 Skill Structure

  • Skills are directories containing required SKILL.md instructions with YAML frontmatter and optional bundled resources; agents expose a terminal-output-to-action step interface.Frontmatter supplies the Skill name and discovery description, while the body provides procedural guidance, code examples, and usage patterns. Optional resources include executable scripts and reference documentation.Source: §D.5 · p. 21, §D.4 · p. 21

D.6 Self-Generated Skills Condition

  • Self-generated Skills use separate creator and fresh solver sessions, replacing curated packs without changing task instructions or transferring the creator’s reasoning.Creators receive only Anthropic’s official skill-creator and must author packs rather than solve the task. Claude Code regenerates packs each trial in a shared sandbox that permits filesystem carryover; Codex and Gemini CLI reuse packs generated once per task–configuration pair.Source: §D.6 · pp. 21–22

D.8 Container Environment / D.9 Inference Configuration

  • Execution controls include fresh self-generated solver sessions, disposable CPU-only task containers, deterministic sampling, and explicit context-management and run-selection policies.Ubuntu 24.04 containers receive 1–4 CPU cores, 2–10 GB memory, and 10 GB storage, and are deleted after each trial. Inference specifies temperature 0, highest available reasoning effort, an 8K-token sliding window dropping oldest turns, and preference for healthy verifier-scored records, using timeouts only as failure backfill.Source: §D.6 · p. 22, §D.8 · p. 24, §D.9 · p. 24

D.6 Self-Generated Skills Condition

  • −8.1 to −11.5 pp versus no Skills contrasts with curated Skills’ +18.2 to +24.8 pp across three dedicated-harness configurations.This diagnostic uses a fixed 87-task denominator, with 258, 258, and 253 scored slots out of 261; missing slots remain coverage gaps. Claude Code used max reasoning effort rather than its baseline’s high, while Codex and Gemini CLI matched their baselines.Source: Table 6 · p. 22, §D.6 · p. 22

D.6.1 Why self-generated Skills underperform: trajectory audit

  • Twelve solver trajectories across ten selected task–configuration pairs underpin the audit, whose per-task interpretations face verifier changes and incomplete artifact visibility.The 3d-scan-calc verifier later accepted the alternative unit interpretation, and unscored records were excluded from causal interpretation. Unreleased pre-baked creator sessions and missing Gemini tool outputs limit visibility, without changing the aggregate direction.Source: §D.6.1 · p. 23
  • 10 of 12 audited pre-baked-family trajectories never accessed generated Skills; other audited failures involved authoring interference or confidently wrong procedural guidance.Claude Code authoring could prevent solver artifact production, while consumed packs could enforce incorrect unit conversions or omit validation cross-checks. The clearest positive case embedded instance-specific verifier targets and fixes, indicating leakage rather than reusable procedure.Source: §D.6.1 · p. 23

D.10 Experiment Orchestration … H Complete Task List

Task-level results show broad but uneven benefits from curated Skills, while design analyses favor small, focused packages and identify repeatable failure modes.

D.10 Experiment Orchestration; H Complete Task List

  • 9,396 selected public result files cover 18 configurations, 87 tasks, two conditions, and three trials per task; Table 19 groups the inventory into eight domains.Main conditions use 256 concurrent trials, with unscored, stale, or rate-limited runs audited and rerun. The inventory table is generated from scripts/taxonomy.csv.Source: §D.10 · p. 24, H Complete Task List · p. 29

E Task-Level Results

  • Curated Skills benefit most task–model pairs, although some regress; easy tasks remain universally solved and hard tasks remain unsolved even with Skills.The solved-task region expands relative to the no-Skills baseline, while the solvable-to-unsolvable transition follows model capability. Results cover all 87 tasks using a fixed three-trial denominator.Source: E Task-Level Results · p. 24, Figure 11 · p. 25, Figure 12 · p. 26

F Skill Design Factors; F.1 Skill Quantity

  • One to three Skills perform best; ≥4 Skills introduces diminishing returns consistent with cognitive overhead and conflicting guidance.These design ablations use the latest 87-task, 18-configuration aggregate and the same fixed three-trial denominator as Table 2.Source: Table 8 · p. 28, F Skill Design Factors · p. 28

F.2 Skill Complexity

  • Focused Skill documentation outperforms comprehensive prose when grouped by total SKILL.md text size.Documentation-length buckets use the 25th, 50th, and 95th percentile cutpoints.Source: Table 9 · p. 28

F.3 Failure Modes when Skills Hurt

  • Skills hurt when heavyweight pipelines crowd out simpler execution, generic recipes displace stronger native strategies, or prescribed solvers exceed agents’ debugging ability.Authors should mark optional steps and provide fast paths, specify applicability boundaries, and supply debugging fallbacks and validation checks.Source: §F.3 · p. 28

F.4 Skill-Author Patterns Audit

  • +67.0 pp mean uplift characterized the 10 highest-∆ tasks audited to extract five Skill-authoring patterns.The patterns recommend executable scripts with calibrated defaults, canonical data sources and parsing quirks, exact verifier-checked file-format constraints, algorithmic invariants, and task-specific description frontmatter. Each is grounded in paired trajectory excerpts released alongside the trajectories.Source: §F.4 · pp. 28–29

G Confidence Interval Calculation

  • 95% binomial (Wald) confidence intervals accompany main-result and Skill-invocation rates; normalized gain remains a point estimate.Intervals use the corresponding rate and selected public result-file count for each model–harness cell. Resolution-rate intervals appear as offset circles within stacked bars, while invocation intervals appear as black diamonds with vertical ranges.Source: G Confidence Interval Calculation · p. 29

I Comprehensive Results Summary … J.2 Qualitative Failure Examples

Across 18 model–harness configurations, curated Skills improve average pass rate by 16.6 percentage points, with every model family improving without a material time cost. Task-level audits illustrate benefits from procedural guidance alongside remaining data-interpretation, completeness, and output-format failures.

I Comprehensive Results Summary

  • +16.6 pp average pass-rate improvement corresponds to 25.5% normalized gain across 18 model–harness configurations using curated Skills.On the fixed 87×3 trial frame per condition, OpenHands + GPT-5.5 leads with Skills at 67.3%; configuration improvements range from +4.1 to +25.7 pp. OpenHands + GLM 5.1 achieves the largest improvement (+25.7 pp), while Gemini CLI + Gemini 3.1 Pro achieves the highest normalized gain (38.7%).Source: Table 10 · p. 29, I Comprehensive Results Summary · pp. 29–30
  • Every model family improves with curated Skills without a material increase in mean agent wall-clock time per task.Figure 14 compares each family’s no-Skills and curated-Skills results in the time–performance plane.Source: Figure 14 · p. 30, I Comprehensive Results Summary · p. 30

I.1 Auxiliary Failure-Mode Snapshot

  • The auxiliary diagnostic snapshot reports condition-level failure modes and pooled configuration-level outcomes, distinguishing failure-rate denominators from failure-mode shares.Failure rates use 4,698 slots per condition, while mode shares use selected non-solved trials; selected public coverage spans all 9,396 slots. Configuration-level outcomes pool both Skills conditions and are ordered by solved rate.Source: Table 11 · p. 30, Table 12 · p. 31

J Task-Level Case Studies

  • +38.9 pp lifts sales-pivot-analysis from 1.9% to 40.7%, illustrating procedural guidance’s contribution in verifier-scored qualitative case studies.These audits supplement aggregate results: step-by-step openpyxl pivot-table guidance replaces unsuccessful manual DataFrame reshaping. Without Skills, missing pivot objects caused 10/23 tests to fail despite correctly populated source data.Source: §J.1 · p. 31, J Task-Level Case Studies · p. 31, §J.2 · p. 32

J.1 Success Case Studies

  • +66.7 pp on flood-risk-analysis and +68.5 pp on sec-financial-report illustrate gains from explicit statistical pipelines and regulatory guidance, respectively.Flood-risk-analysis rises from 1.9% to 68.5% with Log-Pearson Type III guidance specifying scipy calls and parameter interpretation. sec-financial-report rises from 0% to 68.5% with SEC EDGAR endpoints, 13F-HR filing structure, and cross-quarter comparison procedures.Source: §J.1 · pp. 31–32
  • +55.6 pp raises manufacturing-fjsp-optimization from 0% to 55.6% when Skills guide constraint-aware scheduling instead of schedules that ignore maintenance.The Skill specifies constraint propagation, objective-function formulation, and OR-Tools solver configuration for machine downtime windows; only a subset of configurations succeeds.Source: §J.1 · p. 32

J.2 Qualitative Failure Examples

  • 7/8 tests passed for earthquake-plate-calculation and 6 of 9 for shock-analysis-supply, yet incorrect boundary data and missing computational components prevented success.Despite correct event metadata and the correct Haversine formula, earthquake-plate-calculation returned 3,562 km instead of 3,878 km, exceeding the ±0.01 km tolerance. shock-analysis-supply created correct workbook structures but omitted Penn World Tables employment/labor data, HP filter optimization, and depreciation-rate computation.Source: §J.2 · p. 32
  • Other failures involved missing gh-repo-analytics output, excess latex-formula-extraction content, and misunderstood exceltable-in-ppt formula dependencies.gh-repo-analytics never created /app/report.json, causing all 8 tests to error at the fixture stage in a pipeline requiring Gitea interaction, repository cloning, and analytics. latex-formula-extraction included headers, yielding 6 entries instead of 5 and violating one-formula-per-line requirements; exceltable-in-ppt passed 6/8 tests but omitted inverse-rate and dependent-cell recomputation, propagating NaNs.Source: §J.2 · p. 32

J.3 Tasks With Largest Skills Impact … M.8 Benchmark Composition

Curated Skills improve fleet resolution rates at broadly unchanged agent time, while token usage and cost vary across configurations. SkillsBench combines reusable procedural packages with deterministic verification, contribution review, and safeguards against task-specific solution leakage.

J.3 Tasks With Largest Skills Impact

  • +67.0 percentage points is the average pass-rate improvement across the ten tasks with the largest Skills impact.These tasks require domain-specific procedural knowledge, including cache behavior, intrusion detection, financial reporting schemas, scientific pipelines, optimization, and artifact conversion.Source: §J.3 · p. 32, Table 13 · p. 33

L.2 Outcome and Runtime Diagnostics; K Skill Invocation Rate

  • 0.8% of selected trials—78 of 9,396—have runtime errors rather than scored results.Runtime errors count as reward 0 in the main aggregate, whereas attempted failures are verifier-scored trials with reward 0. Skill-invocation counts include only task-specific Skills under environment/skills, excluding harness-bundled Skills.Source: K Skill Invocation Rate · p. 33, Table 16 · p. 34, Table 17 · p. 35

L Token, Cost, and Time Efficiency; L.1 Token Usage and Cost by Configuration; L.3 Cost-Performance Tradeoff

  • 67.3% with-Skills pass rate at $1.67 per trial makes OpenHands + GPT-5.5 stronger and cheaper than OpenHands + Claude Opus 4.8 (54.1% at $14.02).The healthy-first three-trial selection provides token counts for all 18 configurations and priced costs for 10, suppressing cells with fewer than 20 usable trials; Claude Code token counts are approximate. Skills increase tokens for some configurations and decrease them for others, while prompt-cache reads can materially affect realized cost.Source: L Token, Cost, and Time Efficiency · pp. 33–34, Table 15 · p. 34, §L.3 · p. 35

L.4 Time–Performance Tradeoff

  • +16.6 percentage points in fleet mean resolution rate come at broadly unchanged agent time with curated Skills.The slower half of the fleet runs faster with Skills; OpenHands + Claude Opus 4.7 drops from 10.0 to 6.5 minutes per task. The 4–9-minute band contains both the strongest and weakest configurations.Source: §L.4 · p. 35, Figure 15 · p. 36

M.1 Skill Definition and Boundary; M.7 Leakage Prevention

  • Skills are portable procedural packages for task classes, with required SKILL.md files and optional resources, not repositories of benchmark-specific answers.Skills must be authored independently of benchmark specifications and exclude task-specific identifiers, constants, expected outputs, test references, and exact solution commands. A Claude Code Agent SDK-based CI validator detects potential leakage, with failed tasks rejected to preserve reusable procedural augmentation.Source: §M.1 · p. 36, §M.7 · p. 38

M.3 Community Submission Model; M.4 Contributing Principles

  • 142 contributors submitted 400 candidate tasks, from which maintainers curated the evaluated set under explicit contribution requirements.Instructions must be human-authored and must not identify which Skills to use; Skills must provide task-class guidance rather than instance-specific solutions. Success criteria require programmatic assertions with sufficient, nonredundant coverage, informative errors, and parametrization where possible.Source: §M.3 · p. 37, §M.4 · p. 37

M.5 Automated Validation; M.6 Human Review

  • Automated validation precedes human review, checking package structure, oracle success, instruction quality, and potential Skill-solution leakage.Human reviewers assess realistic data and workflows, expert-quality reference solutions, reusable and consistent Skills, and defenses against shortcut solutions. Reviewers also run multiple agents with and without Skills to confirm meaningful efficacy signals.Source: §M.5 · p. 37, §M.6 · pp. 37–38

M SKILLSBENCH Construction Details; M.2 Task Package; M.8 Benchmark Composition

  • 87 tasks across 8 domains use self-contained packages comprising instructions, Docker environments, reference solutions, and deterministic verifiers.Instructions must be solvable by knowledgeable humans without paired Skills, and reference solutions must pass verification with 100% success before acceptance. Difficulty reflects estimated unaided completion time for median domain specialists, supplied by contributors and reviewed by domain-qualified maintainers.Source: §M.2 · pp. 36–37, §M.8 · p. 38

M.9 Domain Taxonomy and Per-Task Mapping … N.4 Scoring

The protocol compares matched no-Skills and curated-Skills runs using deterministic verification and task-macro scoring across 87 tasks. Small domain slices remain descriptive, and normalized gains are interpreted alongside absolute pass-rate changes.

M.9 Domain Taxonomy and Per-Task Mapping

  • Eight domains organize the task inventory, but Cybersecurity and Media & Content Production remain descriptive slices because each contains fewer than eight tasks.Table 19 lists all 87 tasks with primary capability and difficulty: C=Core (<60 min), X=Extended (1–4 h), and E=Extreme (>4 h). Capability axes are Reasoning, Agentic Coding, Multimodal, Tool Use, and Search & Research; scripts/distribution.py generates the mapping from scripts/taxonomy.csv to keep it synchronized with figures.Source: §M.9 · p. 38, Table 19 · p. 39

N Experimental Protocol Details

  • The experimental appendix separates evaluation details from benchmark construction; results use model–harness units because harnesses couple model behavior, context construction, tool interfaces, and execution control.This separation makes the protocol, agent configurations, and scoring choices easier to audit.Source: N Experimental Protocol Details · p. 42, §N.1 · p. 42

N.1 Model–Harness Configurations

  • Four harnesses are evaluated: OpenHands, Claude Code, Codex CLI, and Gemini CLI.All models use temperature 0; Appendix Table 5 records model identifiers, harness versions, and selected result counts. Claude models’ training awareness of the Agent Skills specification may affect their interpretation of Skill-formatted instructions.Source: §N.1 · p. 42

N.2 Skills Conditions

  • Matched no-Skills and curated-Skills conditions differ in Skill access, while task environments, input data, resource limits, and verifiers remain identical.Curated runs receive the complete environment/skills/ directory, including instructions, examples, scripts, and references; no-Skills runs receive none. Self-generated Skills are evaluated separately on three dedicated-harness configurations, outside the two-condition aggregate.Source: §N.2 · p. 42, §N.3 · p. 42

N.3 Execution Protocol

  • Curated Skills load before task instructions through each harness’s native mechanism, and deterministic assertions score the agent’s final answer or artifact.The agent interacts with a containerized environment until submission, after which the verifier writes a pass/fail result.Source: §N.3 · p. 42
  • Selected results require a public result file, complete trajectory metadata, and a healthy verifier-scored outcome; incomplete runs are rerun rather than counted as outcomes.Timeouts serve only as failure backfill when healthy replacements are unavailable. Stale, rate-limited, and otherwise unscored runs count as incomplete coverage.Source: §N.3 · p. 42

N.4 Scoring

  • 87 tasks contribute equally to task-macro pass rate, with each task–model pair scored using three selected public trials per condition.Following Terminal-Bench [Merrill et al., 2026], scoring first averages deterministic verifier rewards in [0, 1] within each task, then averages across tasks.Source: §N.4 · p. 42
  • Normalized gain measures progress toward perfect performance but can overstate small absolute gains near the ceiling, requiring interpretation alongside absolute pass-rate deltas.Absolute improvement is PassRate(curated) − PassRate(no Skills). Figure 1 confidence intervals use the binomial calculation described in Appendix G.Source: §N.4 · p. 42

Related papers

Loading 2602.12670v4…