Source-linked AI summary
SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?
Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, Lijie Hu
TL;DR
Real-world evidence for the utility of agent skills in software engineering remains limited because existing benchmarks do not isolate skill augmentation in requirement-driven, repository-grounded workflows. SWE-Skills-Bench addresses this gap with 49 public skills, approximately 565 fixed-commit GitHub tasks, and deterministic execution-based verification under paired skill conditions. Across the benchmark, 39 of 49 skills yield no pass-rate improvement and the average gain is +1.2%, while a small specialized subset helps substantially and some skills interfere with project context.
Problem
Existing benchmarks lack a software-engineering evaluation that isolates agent-skill utility in repository-grounded workflows with explicit requirements.
Method
SWE-Skills-Bench pairs 49 public SWE skills with fixed-commit GitHub projects and requirement documents, then maps acceptance criteria to deterministic tests for paired evaluation.
Results
39 of 49 skills produce ΔP = 0, while average pass-rate improvement is +1.2%; seven specialized skills gain up to +30%, and three degrade performance by up to −10%.
Takeaways & Limitations
Skill utility is highly domain-specific and context-dependent, favoring targeted skill design over blanket adoption.
Takeaways & Limitations
All experiments use a single agent configuration, Claude Code with Claude Haiku 4.5, so utility across other foundation models remains to be evaluated.
Abstract
from arXiv · showhide
Agent skills, structured procedural knowledge packages injected at inference time, are increasingly used to augment LLM agents on software engineering tasks. However, their real utility in end-to-end development settings remains unclear. We present SWE-Skills-Bench, the first requirement-driven benchmark that isolates the marginal utility of agent skills in real-world software engineering (SWE). It pairs 49 public SWE skills with authentic GitHub repositories pinned at fixed commits and requirement documents with explicit acceptance criteria, yielding approximately 565 task instances across six SWE subdomains. We introduce a deterministic verification framework that maps each task's acceptance criteria to execution-based tests, enabling controlled paired evaluation with and without the skill. Our results show that skill injection benefits are far more limited than rapid adoption suggests: 39 of 49 skills yield zero pass-rate improvement, and the average gain is only +1.2%. Token overhead varies from modest savings to a 451% increase while pass rates remain unchanged. Only seven specialized skills produce meaningful gains (up to +30%), while three degrade performance (up to -10%) due to version-mismatched guidance conflicting with project context. These findings suggest that agent skills are a narrow intervention whose utility depends strongly on domain fit, abstraction level, and contextual compatibility. SWE-Skills-Bench provides a testbed for evaluating the design, selection, and deployment of skills in software engineering agents. SWE-Skills-Bench is available at https://github.com/GeniusHTX/SWE-Skills-Bench.
1 Introduction
SWE-Skills-Bench addresses the lack of real-world SWE benchmarks that isolate agent-skill utility by pairing repository-grounded requirements with deterministic evaluation. Its findings indicate that skill injection usually provides little benefit, though specialized skills can substantially help or interfere with project context.
- Motivation: Existing benchmarks do not jointly evaluate agent skills, multi-file software projects, and requirement satisfaction with deterministic verification.TerminalBench lacks a skill-augmentation condition, while HumanEval and BigCodeBench omit multi-file project context; SkillsBench is broader than SWE.
- Benchmark: SWE-Skills-Bench curates 49 real-world SWE skills and approximately 565 task instances on fixed-commit GitHub projects in containerized environments.Tasks are sourced from public skill repositories and evaluated under controlled project conditions.
- Evaluation: The benchmark translates each requirement into executable unit tests that deterministically verify fulfillment with and without skill injection.This requirement-driven harness avoids subjective LLM-as-judge evaluation.
- Findings: 39 of 49 skills produce ΔP = 0, while average pass-rate improvement is only +1.2%.Most injected skills therefore do not change task-level success rates.
- Findings: Seven specialized skills achieve gains up to +30%, whereas three skills degrade performance by up to −10% when version-specific guidance conflicts with project frameworks.The reported results characterize skill utility as domain-specific and context-dependent.
2 Related Benchmarks & Datasets
Prior benchmarks either evaluate realistic SWE tasks without isolating skill augmentation or evaluate skills across domains without focusing on repository-grounded requirements. SWE-Skills-Bench is positioned as a benchmark specifically designed to combine SWE realism, skill conditions, and requirement-driven verification.
- SWE Benchmarks: SWE real-world benchmarks use realistic project-level tasks and execution-based verification but do not isolate the marginal benefit of procedural skill documents.SWE-Bench Verified and TerminalBench provide realistic evaluation settings without controlled skill comparisons.
- SWE Benchmarks: Code-generation benchmarks mainly test self-contained functions and omit multi-file reasoning, dependency management, and end-to-end SWE workflows.HumanEval is cited as a function-level benchmark rather than a repository-grounded development evaluation.
- Skill Benchmarks: SkillsBench compares agents across skill conditions but is not SWE-specific and is not centered on explicit requirements in repository-grounded workflows.SWE-Skills-Bench addresses this gap with a requirement-driven benchmark focused exclusively on software engineering.
3 SWE-Skills-Bench Construction
SWE-Skills-Bench constructs repository-grounded task instances by curating evaluable skills, pairing them with fixed-commit projects and explicit requirements, and deriving deterministic verifiers. The resulting setup supports paired evaluation with and without injected skills.
- Skill Curation: 84,192 public skills are filtered through category selection, semantic filtering, and feasibility screening to produce 49 SWE skills.The selected categories align with software-engineering workflows and unit-test evaluation.
- Pipeline: The construction pipeline selects skills, pairs them with authentic task instances, and verifies stated requirements through deterministic verifiers.These stages address which skills to benchmark, how to create repository-grounded tasks, and how to verify fulfillment.
- Task Generation: Each curated skill is paired with approximately 10 task instances using an authentic open-source project whose repository is pinned at a fixed commit.Projects are matched by technology stack and prepared in Docker containers for reproducibility.
- Task Generation: Each requirement document specifies background, objectives, file operations, and deterministic acceptance criteria tailored to the target repository and skill conditions.The standardized template is intended to reduce ambiguity and clarify success conditions.
- Task Formulation: A task instance is represented as (R, E, P, S), and the agent must produce code, configuration, or execution artifacts satisfying P in repository R and environment E.S is optionally injected, enabling controlled with-skill versus without-skill comparisons.
- Verification: Every acceptance criterion is mapped to a deterministic verifier, establishing traceability from requirements to test verdicts.The evaluation metrics include pass rates, token costs, token overhead ratio ρ, and cost efficiency CE.
4 Results of SWE-Skills-Bench
Across controlled with-skill and without-skill evaluations, skill injection produced heterogeneous outcomes: modest aggregate gains, substantial token variation, meaningful benefits for a small subset, and performance regressions from context interference.
- Evaluation setup: The evaluation compares use-skill and no-skill conditions with autonomous skill discovery in containerized projects.Each task is evaluated under both conditions, with SKILL.md present only when skills are enabled.
- Aggregate results: Average pass rate rose 1.2% from 89.8% to 91.0%, while average token consumption increased 10.5% across 49 skills.The per-skill results are highly heterogeneous beneath these aggregate averages.
- Pass-rate utility: 39 of 49 skills produced ΔP = 0, and the average skill utility delta was +1.2%.For 24 skills, both conditions achieved 100% pass rates, while 15 had identical imperfect rates.
- Token cost: Among skills with ΔP = 0, token overhead ρ ranged from −77.6% to +450.8%, showing that reasoning-path changes need not alter final correctness.Eight perfect-score skills used fewer tokens with injection, whereas 16 used more; ρ and ΔP showed no consistent correlation.
- Positive gains: Seven skills achieved positive gains from +7.1% to +30.0%, with risk-metrics-calculation reaching ΔP = +30.0% and ρ = −34.8%.By contrast, tdd-workflow improved pass rate by +7.1% while incurring 78.6% token overhead.
- Context interference: Three skills degraded performance: springboot-tdd by −10.0%, linkerd-patterns by −9.1%, and django-patterns by −9.1%.The authors attribute these regressions to surplus context, over-fitted templates, and competition for finite context-window space.
- Context interference: The linkerd-patterns skill caused errors by anchoring on v1beta1 and HTTP/1, fabricating rules/metricsServers, and appending an unrequested NetworkPolicy.Without the skill, the agent selected v1beta3, gRPC, and standard meshTLS.serviceAccounts; the near-match template instead polluted task interpretation.
- Design implication: The results support abstract guidance patterns over concrete, opinionated templates with hard-coded values that may not transfer to target tasks.This implication follows from the linkerd-patterns failure despite the skill containing relevant domain content.
5 Discussion & Future Directions
The benchmark is presented as an evolving foundation for studying how procedural skill injection affects software engineering agents. Future work targets generalization across models and agent scaffolds, dynamic skill use, and empirically grounded skill-design principles.
- Discussion & Future Directions: The benchmark currently covers only a fraction of the design space, with substantial extensions underway.The authors describe the results as a snapshot of an actively evolving research program.
- Discussion & Future Directions: All experiments use Claude Code with Claude Haiku 4.5, limiting conclusions about model–skill interactions.Planned multi-model evaluation will vary model scale, training data composition, and architecture.
- Discussion & Future Directions: The findings may depend on the agent scaffold governing tool use, planning, and context management.Different frameworks may allocate context budgets, retrieve long skill documents, and structure reasoning differently.
- Discussion & Future Directions: Skill form may matter as much as content: concrete templates with hard-coded values can anchor agents on non-transferable specifics.The paper proposes studying granularity, abstraction level, and structural organization to guide skill authors.
- Discussion & Future Directions: The current framework assumes one pre-placed skill per task, leaving retrieval, composition, and ambiguity-robust selection unevaluated.Dynamic skill selection and multi-skill interaction are identified as important benchmark extensions.
System prompt
The system prompt instructs a professional test engineer to generate automated tests that determine whether an agent completed a programming task. The verification process emphasizes behavioral, discriminative, and complete testing, while the benchmark also uses prompts for requirement generation and requirement-driven verification.
- System prompt: The system prompt assigns a professional test engineer to generate automated test suites for completed programming tasks.The generated suite is executed after the agent finishes to determine correctness.
- System prompt: Tests must distinguish genuinely completed tasks from superficially plausible outputs through behavioral verification.The prompt rejects reliance solely on static checks such as keyword matching or file existence.
- System prompt: Figure 6 presents the prompt used to generate requirement-driven verification tests.The benchmark converts repository-specific requirements and acceptance criteria into deterministic tests.