PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?

Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, Lijie Hu

arXiv:2603.15401v1cs.SEcs.AI

TL;DR

Whether agent skills improve real-world software development remains unclear. SWE-Skills-Bench compares agents with and without skills on requirement-driven GitHub tasks using deterministic execution checks. Skill injection yields limited pass-rate gains averaging +1.2%, while token overhead varies widely even when success rates remain unchanged.

  • Problem

    Existing benchmarks provide limited evidence on whether agent skills improve requirement satisfaction in real-world software development workflows.

    Source: §1 · p. 2

  • Method

    SWE-Skills-Bench pairs public skills with fixed-commit GitHub projects and requirements, comparing agents with and without skills using deterministic execution checks tied to acceptance criteria.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Skill injection yields limited pass-rate gains, averaging +1.2%, while token overhead varies widely even when task success rates remain unchanged.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Skill usefulness is domain-specific, and concrete templates risk anchoring agents to details that do not transfer to the target task.

    Source: §5 · pp. 9–10

    All experiments use a single agent configuration, Claude Code with Claude Haiku 4.5.

    Source: §5 · p. 10

Abstract

from arXiv · show

Agent skills, structured procedural knowledge packages injected at inference time, are increasingly used to augment LLM agents on software engineering tasks. However, their real utility in end-to-end development settings remains unclear. We present SWE-Skills-Bench, the first requirement-driven benchmark that isolates the marginal utility of agent skills in real-world software engineering (SWE). It pairs 49 public SWE skills with authentic GitHub repositories pinned at fixed commits and requirement documents with explicit acceptance criteria, yielding approximately 565 task instances across six SWE subdomains. We introduce a deterministic verification framework that maps each task's acceptance criteria to execution-based tests, enabling controlled paired evaluation with and without the skill. Our results show that skill injection benefits are far more limited than rapid adoption suggests: 39 of 49 skills yield zero pass-rate improvement, and the average gain is only +1.2%. Token overhead varies from modest savings to a 451% increase while pass rates remain unchanged. Only seven specialized skills produce meaningful gains (up to +30%), while three degrade performance (up to -10%) due to version-mismatched guidance conflicting with project context. These findings suggest that agent skills are a narrow intervention whose utility depends strongly on domain fit, abstraction level, and contextual compatibility. SWE-Skills-Bench provides a testbed for evaluating the design, selection, and deployment of skills in software engineering agents. SWE-Skills-Bench is available at https://github.com/GeniusHTX/SWE-Skills-Bench.

1 Introduction

SWE-Skills-Bench isolates agent skills’ marginal utility through requirement-driven, paired evaluation on real-world software projects. Its findings show limited average pass-rate gains, with utility depending on domain specialization and compatibility with project context.

Benchmark gap

  • 16 of 84 tasks in SkillsBench [9] cover SWE; its focus is cross-domain skill efficacy rather than requirement satisfaction in real-world development.No existing benchmark evaluates SWE skills in real-world software development scenarios. TerminalBench [10] lacks skill augmentation, while HumanEval [11] and BigCodeBench [12] target self-contained function completion without multi-file context or skill augmentation.Source: §1 · p. 2

Benchmark

  • 49 public SWE skills underpin approximately 565 task instances evaluated on fixed-commit GitHub projects in containers under controlled with-skill versus without-skill conditions.Agent skills are structured markdown packages encoding procedural knowledge [5] [6] [7] [8], injected into the context window as reference documents without model modification or external retrieval.Source: §1 · p. 2

Requirement-driven test harness

  • The requirement-driven harness translates explicit acceptance criteria into executable unit tests, tracing requirements to deterministic verdicts under both skill conditions without LLM-as-judge evaluation.Each task’s requirement document defines its scope and acceptance criteria.Source: §1 · pp. 2–3

Empirical findings

  • 39 of 49 skills produce ∆P = 0, and the average pass-rate improvement is only +1.2%.These findings favor targeted skill design over blanket adoption.Source: §1 · p. 3
  • 7 specialized skills deliver gains up to +30%, while three skills produce negative deltas up to −10% when version-specific conventions conflict with project frameworks.Beneficial specializations include financial risk formulas, cloud-native traffic management, and GitLab CI patterns.Source: §1 · p. 3
  • −78% to +451% is the token overhead ratio ρ range among skills with zero pass-rate delta, showing that token overhead is decoupled from correctness.Skills can reshape the agent’s reasoning path without improving outcomes.Source: §1 · p. 3

2 Related Benchmarks & Datasets

Existing SWE benchmarks do not isolate the benefit of procedural skills, while SkillsBench [9] is not SWE-specific. SWE-Skills-Bench addresses this gap with explicit requirements, fixed-commit repositories, and deterministic execution-based verification.

SWE-related Benchmarks

  • SWE-Bench Verified and TerminalBench [10] evaluate realistic tasks with execution-based verification but do not isolate the benefits of injecting procedural skill documents.SWE-Bench Verified contains 500 human-validated instances from 12 Python repositories. TerminalBench [10] covers 200 containerized CLI tasks and informs this paper’s evaluation setup.Source: §2 · p. 3
  • 164 function-level challenges in HumanEval [11] assess self-contained code generation rather than multi-file reasoning, dependency management, or end-to-end SWE workflows.Code generation benchmarks generally evaluate algorithmic or snippet-level problems without full project context.Source: §2 · p. 3

Skills Benchmarks

  • SkillsBench [9] compares skill conditions but is not SWE-specific; SWE-Skills-Bench pairs skills with fixed-commit repositories, explicit requirements, and deterministic execution-based verification.Software engineering forms only a limited subset of SkillsBench [9], which is not designed around requirement satisfaction in repository-grounded workflows.Source: §2 · p. 3

3 SWE-Skills-Bench Construction

SWE-Skills-Bench pairs 49 curated SWE skills with fixed-commit GitHub projects to create 565 task instances, then verifies requirements through deterministic tests. Paired evaluation compares agent performance with and without the skill document.

3.2 Task Instance Generation

  • 565 task instances pair curated skills with fixed-commit GitHub projects, containerized environments, and repository-specific requirements containing explicit acceptance criteria.Approximately 10 instances are generated per skill, using projects whose technology stacks match the skill’s domain. Requirements follow a standardized template: Background, Requirement, File Operations, and Acceptance Criteria.Source: Figure 3 · p. 4, §3 · p. 4, §3.2 · p. 5
  • Paired evaluation controls skill exposure through file presence, while requirement documents never mention the skill.Preparation removes the repository’s existing .claude/skills directory, and the skill document is copied into ~/.claude only for the skill-enabled condition. The agent automatically detects and integrates skills present in that environment.Source: Figure 3 · p. 4, §3.2 · p. 5

3.1 Skill Curation

  • 49 skills across six SWE categories survive category selection, semantic filtering, and repository-feasibility screening.Filtering excludes generative or subjective skills and candidates with prohibitively large repositories or costly setup. The categories are Deployment & DevOps (13), Analytics & Monitoring (12), API Development (10), Data Science & ML (9), Security & Testing (4), and Developer Tools (1).Source: §3.1 · p. 4

3.3 Requirement-driven Verification

  • Requirement-driven verification maps every acceptance criterion to deterministic tests that execute produced code and check concrete outputs or structures rather than keyword heuristics.A fixed test-engineer prompt uses requirements and repository metadata to generate representative and edge-case pytest scenarios. It also requires a minimum number of test cases and per-test docstrings.Source: §3.3 · p. 5

3.4 Task Formulation

  • Each task is formulated as (R, E, P, S): a fixed-commit repository, containerized environment, natural-language requirements, and optional skill document.Claude Code must produce code changes, configuration files, or execution artifacts that satisfy P within repository R and environment E.Source: §3.4 · p. 5

4 Results of SWE-Skills-Bench

Skill injection produces a small average pass-rate gain while increasing token consumption. Benefits concentrate in a few skills, while mismatched guidance can degrade performance through context interference.

4.1 Experimental Setup

  • Claude Code with Claude Haiku 4.5 runs paired skill/no-skill evaluations in CPU-only Ubuntu 24.04 Docker containers with per-task resource limits.For skill-enabled runs, SKILL.md is placed in the project root, where the agent discovers and applies it autonomously without explicit instruction.Source: §4.1 · pp. 6–7

4.2 Evaluation Metrics

  • Pass Rate, Skill Utility Delta, Token Cost, and Cost Efficiency assess task success, marginal skill benefit, and inference overhead.Pass Rate is the primary metric; token consumption is averaged by condition and compared through an overhead ratio. Cost Efficiency measures success-rate improvement per unit of relative token increase.Source: §4.2 · p. 7

4.3 Evaluation Results

  • +1.2% average pass-rate improvement, from 89.8% to 91.0%, accompanies a 10.5% increase in average token consumption across 49 skills.Aggregate gains conceal substantial variation between individual skills.Source: §4.3 · p. 7
  • 39 of 49 skills produce no pass-rate improvement, including 24 with 100% pass rates both with and without skills.The other 15 retain identical but imperfect pass rates. The authors suggest their bottlenecks involve deeper capability gaps rather than simply missing domain knowledge.Source: §4.3 · p. 7
  • −77.6% to +450.8% token overhead occurs among skills with unchanged perfect pass rates, demonstrating that token costs can vary without correctness gains.Among these 24 skills, python-resilience has ρ = −77.6% and service-mesh-observability has ρ = +450.8%; eight reduce tokens and sixteen increase them. Across all 49 skills, token overhead and pass-rate gains show no consistent correlation.Source: §4.3 · p. 8
  • +30.0% pass-rate improvement and ρ = −34.8% make risk-metrics-calculation the strongest of seven improving skills, whose gains range from +7.1% to +30.0%.By contrast, tdd-workflow gains +7.1% with 78.6% token overhead and CE = 0.09. Its checklist improves coverage of overlooked deliverables but requires additional verification and follow-through.Source: §4.3 · p. 8
  • −10.0%, −9.1%, and −9.1% pass-rate changes occur for springboot-tdd, linkerd-patterns, and django-patterns, respectively.The authors attribute these regressions to comprehensive guidance conflicting with focused task requirements. Surplus context expands unnecessary design choices, encourages rigid template-following, and competes with task and codebase information.Source: §4.3 · p. 8
  • Near-matching Linkerd templates trigger surface anchoring, hallucinated API fields, and concept bleed, motivating abstract guidance rather than hard-coded templates.Without skills, the agent correctly uses v1beta3 and gRPC; injection instead anchors it on v1beta1 and HTTP/1. It also fabricates rules/metricsServers and appends an unrequested NetworkPolicy, conflating Linkerd authorization with Kubernetes network controls.Source: Figure 5 · p. 9, §4.3 · p. 9

5 Discussion & Future Directions

SWE-Skills-Bench’s current findings highlight domain-specific skill utility and context-interference risks, but cover only part of the design space. Planned extensions address models, agent scaffolds, skill design, and dynamic selection and composition.

Multi-model evaluation

  • All experiments use Claude Code with Claude Haiku 4.5, leaving generalization across foundation models untested.Stronger models may find skills redundant, while weaker models may struggle to use injected context. Planned evaluations will separate model capability from skill-induced gains and examine cost–performance trade-offs.Source: §5 · p. 10

Diverse agent scaffolds

  • Agent scaffolds may mediate skill utility through context budgeting, retrieval strategies, and reasoning structure, motivating evaluation beyond the framework used in this study.Planned comparisons include open-source and proprietary frameworks, such as SWE-agent, OpenHands, and Aider, to test whether the findings generalize.Source: §5 · p. 10

Skill design principles

  • Opinionated templates with hard-coded parameters risk anchoring agents to inappropriate specifics, motivating research on skill granularity, abstraction, and structural organization.Abstract guidance may transfer more robustly. Comparing modular sections with monolithic documents could help establish empirically grounded guidelines for skill authors.Source: §5 · p. 10

Dynamic skill selection and composition

  • The current one-skill-per-task setup leaves dynamic retrieval and composition untested, motivating evaluation of retrieval accuracy, multi-skill interactions, and selection under ambiguity.The relevant skill is currently pre-placed in the project; realistic deployments require agents to select from large libraries or compose multiple skills at inference time.Source: §5 · p. 10

System prompt

The system prompt assigns a professional test engineer to generate automated suites that evaluate an AI agent’s programming-task completion. It requires behavioral verification, complete acceptance-criteria coverage, and non-trivial assertions.

System prompt

  • Automated test suites must distinguish genuine task completion from superficially plausible outputs by executing the agent’s code and verifying its actual output.Verification must not rely solely on static checks such as keyword matching or file existence.Source: System prompt · p. 12
  • Tests must cover every acceptance criterion, including boundary conditions and error handling.Source: System prompt · p. 12
  • Assertions must verify the correctness of values, data structures, and logic rather than merely checking their presence.Source: System prompt · p. 12

Related papers

Loading 2603.15401v1…