PaperlayerView exact revision on arXiv

Source-linked AI summary

Signal or Noise? A Benchmark Study of Agent Skills in Web Development

Ziyue Yang, Fan Ding

arXiv:2608.23067v1cs.CL

TL;DR

The paper asks whether injecting a matched Agent Skill improves realistic WebDev tasks enough to justify its recurring prompt cost. It introduces WebDev-Skills-Bench with matched controls and component ablations, finding that target injection generally lowers performance and raises token use, with effects depending on model, task, and Skill-project pairing.

  • Problem

    Existing benchmarks provide limited evidence about whether a matched Skill should be injected, especially because they do not separate Skill content from prompt-length effects.

  • Method

    WebDev-Skills-Bench evaluates 31 public Skills on 50 Web-Bench projects and 1,000 ordered tasks using matched conditions, workspace-aware injection, and leave-one-out component ablations.

  • Results

    Target Skill injection reduced mean ∆Pass@2 by 1.3 to 4.2 pp across four models, lowered task completion depth, and increased token cost by 72% to 394%.

  • Takeaways & Limitations

    Skill value is conditional on the particular Skill-project-model triple, so injection is a per-deployment routing decision rather than a configuration default.

  • Takeaways & Limitations

    Sonnet replicate variability makes individual pair-level estimates require caution, although the N=3 panel keeps model-level means stable.

Abstract

from arXiv · show

Agent Skills are reusable procedural modules that are increasingly injected into coding-agent sessions to encode framework conventions, anti-patterns, and reusable tools. However, because each injected Skill expands the prompt of every query, an effective Skill benchmark must determine not only whether an agent can solve a task, but whether the Skill should have been injected at all. We introduce WebDev-Skills-Bench and use it for a controlled empirical study of 31 public WebDev Skills on 50 Web-Bench projects and 1,000 ordered tasks. The benchmark compares four matched conditions, including a length-matched irrelevant control and leave-one-out component ablations. To isolate Skill effects from prompt-length artifacts, we place only SKILL.md in the prompt while mounting auxiliary files into the agent workspace. Across four models, target Skill injection reduces mean Pass@2 by 1.3% to 4.2%, lowers task completion depth, and increases token cost by 72% to 394%, with gains in only 17% to 36% of Skill-project pairs. Length-matched controls reveal two failure modes: some models are length-distracted, where an equally long irrelevant Skill reproduces most of the loss, while others are content-misled, where prompt length is neutral but Skill content still lowers Pass@2 by 1.1% to 1.4%. Further analysis shows that losses concentrate on easy early tasks, Skill rankings transfer weakly across models, and anti-pattern rules outperform example-heavy content within helpful Skills. These findings recast a matched Skill as a hypothesis about a particular Skill-project-model triple rather than a portable asset, reframing injection as a per-deployment routing decision and making length-matched controls and per-model audits a minimum standard for Agent-Skill evaluation.

1 Introduction

Agent Skills are persistent procedural prompts whose evaluation must account for both task performance and the cost of injecting them. WebDev-Skills-Bench addresses this gap with matched controls and finds that target Skills usually reduce performance while increasing token use.

  • Benchmark motivation: 31 public WebDev Skills were evaluated on 50 Web-Bench projects and 1,000 ordered tasks using four matched conditions.The conditions include no Skill, a target Skill, a length-matched irrelevant Skill, and leave-one-out component ablations.
  • Key findings: Target injection produced negative mean ∆Pass@2, reduced task completion depth, and increased token cost across all four models.Mean ∆Pass@2 ranged from −1.3 to −4.2 pp, while token cost rose by 72% to 394%.
  • Key findings: Only 17% to 36% of Skill-project pairs gained from target injection.The results therefore characterize benefit as a minority outcome rather than a general property of matched Skills.
  • Mechanisms: Length-matched controls distinguish length-distracted models from content-misled models.Sonnet and Qwen reproduced most losses with an equally long irrelevant Skill, whereas GPT-5.1 and DeepSeek showed content-driven losses despite neutral length effects.
  • Implications: The study reframes Skill injection as a per-deployment routing decision requiring length-matched controls and per-model audits.The benchmark and its outputs are open-sourced to support this evaluation approach.

2 Related Work

Prior Skill benchmarks disagree about average benefit, while WebDev benchmarks generally hold prompts fixed. This study positions marginal Skill effects as a distinct evaluation target that separates Skill content from prompt-length effects.

  • Agent Skills: Agent Skills package persistent procedural knowledge as Markdown and optional workspace resources injected across every session query.They encode conventions, anti-patterns, and reusable tools as a behavioral prior.
  • Skill benchmarks: Existing Skill benchmarks report conflicting average outcomes across domains and task settings.SkillsBench reports a +16.2 pp gain, whereas SWE-Skills-Bench finds 39 of 49 Skills with zero pass-rate improvement.
  • WebDev benchmarks: WebDev benchmarks span functional generation, design-to-code, and autonomous web agents while keeping prompts fixed.Examples include Web-Bench, ArtifactsBench, WebArena, and Mind2Web.
  • Evaluation gap: WebDev-Skills-Bench measures the marginal effect of a matched Skill rather than absolute model capability.This target is orthogonal to fixed-prompt capability benchmarks.

3 Benchmark Design

WebDev-Skills-Bench uses reproducible, sequential Web-Bench tasks and controlled Skill conditions to estimate marginal injection effects. Its design fixes the task environment while separating target content, prompt length, and Skill components.

  • Task corpus: The corpus contains 50 Web-Bench projects across 11 stack categories and 1,000 sequentially dependent tasks.Each project contributes 20 ordered tasks tested with deterministic Playwright tests.
  • Skill suite: The 31 third-party Skills span roughly 1.2K–22K characters in SKILL.md length.This range supports byte-matched controls across Skills of different sizes.
  • Injection protocol: Only SKILL.md is injected into the prompt, while auxiliary directories are mounted in the agent filesystem.This makes prompt length depend on SKILL.md alone and keeps the length-matched control tractable.
  • Experimental conditions: C0, C1, C2, and C3 respectively measure baseline performance, target-Skill utility, length effects versus content effects, and structural component effects.C2 defines ∆Length = C2 − C0 and ∆Content = C1 − C2; C3 ablates positive rules, anti-patterns, or example code.
  • Reporting: Table 1 reports model-level C1−C0 effects on 117 core pairs, with bootstrap intervals and relative total-token overhead ρ.The table compares model-wise effects under target injection against the native baseline.
  • Metrics: The study reports ∆Pass@1, ∆Pass@2, task completion depth, and relative token overhead with 95% paired-bootstrap intervals.Task Completion Depth is the longest consecutive Pass@2 prefix in each 20-task chain.

4 Results

Across four models, target Skill injection produces negative average effects despite a positive-gain minority, with losses concentrated on easy tasks and arising through distinct length and content mechanisms. Effects vary sharply across models and show that helpful Skills depend on selective components rather than examples by default.

  • 4.1 Average Skill injection does not justify its token cost: −1.3 to −4.2 pp mean ∆Pass@2 accompanies lower Task Completion Depth and 72% to 394% higher token cost across all four models.The negative average is not explained by pass rate alone: Task Completion Depth falls for every model, while token cost rises in parallel.
  • 4.1 Average Skill injection does not justify its token cost: 17% to 36% of Skill-project pairs gain, so the negative average coexists with a positive-gain tail on every model.Sonnet wins on 30% of pairs, GPT-5.1 on 35%, DeepSeek on 36%, and Qwen on 17%.
  • 4.2 The negative effect concentrates on easy tasks: −4.0 to −10.7 pp easy-task degradation occurs on every model, while moderate and challenging tasks show no consistent loss.The largest reliable losses occur on tasks the model already handles correctly, supporting reporting by chain position.
  • 4.2 The negative effect concentrates on easy tasks: Retry lock-in can turn recoverable early mistakes into chain-terminating failures by fixing structural choices and reducing self-repair flexibility.Web-Bench’s two-attempt budget often recovers simple first-attempt mistakes when the model can vary choices between attempts.
  • 4.3 The negative average has two distinct mechanisms: −3.3 pp and −3.5 pp length terms largely explain Sonnet’s and Qwen’s losses, whereas GPT-5.1 and DeepSeek show near-zero length terms and −1.1 pp and −1.4 pp content terms.The decomposition separates length distraction from content misalignment, implying different mitigations for different models.
  • 4.4 Cross-model contradictions caution against static Skill rankings: Near-zero cross-model correlations and 74% mixed-sign pairs show that Skill utility transfers poorly across model backends.Only 1% of pairs gain on all four models and 4% lose on all four; one pair spans a 55 pp model swing.
  • 4.5 In helpful Skills, anti-patterns are the most cost-effective slice: Anti-patterns are the only directionally reliable slice, while example code averages −0.7 pp and removing it saves 34,482 input tokens per run.Examples are strongly model-dependent: they help DeepSeek and Qwen but hurt Sonnet by −15.3 pp.

5 Benchmark Implications

The benchmark implications treat Skill injection as an empirical routing decision rather than a session-wide default. Deployment should account for model, project, task position, prompt length, and whether benefits survive matched controls.

  • Benchmark injection as an opt-in decision: Unconditional injection is small-and-negative on every model and raises token cost by 72% to 394%, supporting opt-in routing by empirical utility thresholds.The recommended decision signal is conditioned on model, project, and possibly task difficulty.
  • Evaluate by chain position, not only by stack: Skill utility should be evaluated by chain position because degradation concentrates on easy initial tasks.A proposed heuristic is to skip Skills early and inject them only when error rates rise.
  • Per-model curation is necessary: Per-model curation is necessary because near-uncorrelated pair effects make single marketplace rankings unreliable across backends.Useful listings should report model-conditioned utility, target stack, prompt length, and survival after a length-matched control.
  • A length-matched control should be the minimum bar: Length-matched controls should be a minimum benchmark requirement because they distinguish prompt-length artifacts from Skill-content effects and expose different mitigations.The workspace-aware protocol makes this control tractable even for multi-file Skills.

6 Conclusion

WebDev-Skills-Bench reframes Skill evaluation around whether injection is warranted, not merely whether an agent can solve a task. Its controls and ablations show that Skill value is conditional on the Skill, project, and model, making injection a routing problem.

  • WebDev-Skills-Bench asks whether a Skill should have been injected at all, rather than only whether an agent can solve a task.
  • Byte-matched controls and slice ablations distinguish a positive tail, easy-task losses, and model-specific length/content mechanisms.
  • A matched Skill is a hypothesis about a particular (Skill, project, model) triple rather than a portable asset.
  • Capturing Skill value is a routing problem: identify the beneficial minority before paying the injection cost.

Limitations

The study’s conclusions are bounded by seed variability, partial and shared C2 measurements, conservative routing coverage, public Skill selection, and the pre-deployment benchmark setting.

  • Across three Sonnet replicates, aggregate C0 and C1 Pass@2 vary by 4.4 and 3.6 pp, comparable to the headline effect size.The authors caution that individual pair-level estimates should therefore be read carefully, although model-level means remain stable with N=3.
  • C2 uses 109 unique length-matched runs across 117 pairs because some prompts are shared across same-project pairs.Cluster bootstrap and full per-pair C2 deduplication remain open follow-ups.
  • Routing is evaluated only on 117 core-tier pairs, limiting claims about off-target Skill deployment.
  • The Skill set comes from prominent public repositories, so closed enterprise Skills or fine-tuned Skill routers may produce different patterns.
  • WebDev-Skills-Bench is a pre-deployment benchmark without live user traffic, human developer interventions, or product-specific acceptance criteria.Web-Bench projects approximate realistic WebDev work but do not reproduce those online conditions.

Both conditions, 1st attempt, fail with:

The figure contrasts two failed first attempts caused by a colliding button label: the no-Skill retry changes the label, while the Skill-anchored retry preserves the collision.

  • The strict getByText('Create Blog') assertion resolves to both the heading and submit button, producing a violation.
  • In the C0 retry, the submit button is relabeled to a structurally distinct string.
  • In the C1 retry, the Skill’s naming convention preserves the operation name, so it still matches the heading as a substring.
  • Injecting S31 raises Pass@2 to 18% on the illustrated pair, while the length-matched control returns to 0%, yielding ∆Content = +18 pp and ∆Length = 0.
  • The successful second attempt adds wrapper text and an explicit visibility rule, satisfying the rendering assertion and advancing Task-1.

C Cross-Model Sign Reversal: lowdb × database-optimizer

The examples illustrate that Skill effects can reverse across models and vary sharply by pair, so aggregate tendencies do not guarantee a consistent deployment outcome.

  • On lowdb × database-optimizer, Sonnet gains +33 pp, DeepSeek and Qwen each lose 22, and GPT-5.1 is approximately unchanged.This 55 pp swing is not explained by baseline difficulty.
  • A separate Sonnet × pull-loading × js-dom-web-components pair shows an empty wrapper failing visibility checks before S31 adds text and an explicit visibility rule.
  • The illustrated +18 pp content effect far exceeds Sonnet’s −0.9 pp model-level average, showing that decomposition describes tendencies rather than per-pair guarantees.
  • The same Skill content can therefore produce opposite-signed effects across models on one core-tier pair.
  • The panel-wide near-zero cross-model correlation means a single Skill ranking cannot capture deployment-specific effects.

D C3 Slice Ablation: Protocol Detail

The C3 leave-one-out protocol decomposes eligible Skill effects by removing positive rules, anti-patterns, or examples while retaining the header. Contributions are pooled and reported by model using matched ablation runs and cell-level statistical tests.

  • Slice definitions: Each SKILL.md is segmented into a retained header, positive rules, anti-patterns, and example code for leave-one-out ablations.Variants remove exactly one removable slice, while the header remains to keep the prompt well-formed.
  • Pair selection: Only Skills with at least two removable slices and qualifying positive gross effects enter the decomposition analysis.The positive-tail selection means the ablation subset’s +5.1 pp gross effect cannot be interpreted as average Skill benefit.
  • Runs and statistics: 180 ablation runs cover 5 pairs, 3 seeds, 3 variants, and 4 models, with inference at the model×project cell level.Per-slice effects use Wilcoxon signed-rank tests and 10,000-resample paired-bootstrap confidence intervals; task-level McNemar tests corroborate direction.
  • Results: Only anti-patterns sit reliably above zero in pooled Pass@2 contribution, while positive rules and examples are null on average.Per-cell spread is wide for positive rules and examples, so pooled means conceal heterogeneous effects.
  • Results: Example code helps DeepSeek, Qwen, and weakly GPT-5.1 but hurts Sonnet, leaving its pooled Pass@2 effect near zero.The figure reports model-specific cell means with paired-bootstrap confidence intervals.

E Released Artifacts

The release provides analysis code, condition routing, Skill provenance, slice definitions, and derived data for reproducing the paper’s tables and figures. Raw evaluation reports and the base task harness are omitted, so analyses are reproducible without byte-for-byte trajectory reruns.

  • Released artifacts: The repository releases analysis code, all condition routing, provenance for 31 Skills, C3 slice definitions, and derived per-pair and per-task CSVs.The CSVs include condition values, pairwise deltas, slice contributions, chain-position data, rankings, and seed-variance tables.
  • Reproducibility: The repository includes a pipeline that reproduces the paper’s tables and figures from the derived CSVs.
  • Scope boundary: Raw per-task reports and the upstream Web-Bench task harness are omitted, preventing byte-for-byte reruns of every agent trajectory.
Loading 2608.23067v1…