PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents

Gen Dong, Yanjie Gao, Liqun Li, Tianyin Xu, Yu Hua, Fan Yang

arXiv:2608.11888v1cs.AI

TL;DR

Existing skill evaluations report mixed effects but provide limited evidence for attributing failures to specific skills and explaining how they alter agent behavior. This study compares skill-guided runs with no-skill or semantically matched references, finding 307 confirmed failures and identifying task-implementation faults and excessive procedure as major causes.

  • Problem

    Prior evaluations show mixed skill effects, while aggregate outcomes and failed runs alone provide limited evidence for attributing harm to a specific skill or explaining trajectory changes.

    Source: §I · p. 1

  • Method

    The study pairs skill-guided executions with no-skill or semantically matched reference runs on the same task, then analyzes differential outcomes and trajectories and builds SKILLTRIAGE for attribution.

    Source: §I · p. 1, A. Study Design · p. 3

  • Results

    307 confirmed failures comprise 125 functional failures and 182 high-confidence efficiency regressions, with Task-Implementation Fault and Excessive Procedure prominent among their respective causes.

    Source: §I · p. 2

  • Takeaways & Limitations

    The findings support treating skill loading as a costed and validated configuration decision, with compatibility checks, cost prediction, and paired correctness-and-cost evaluation.

    Source: §IX · p. 11

    Findings may depend on the task distribution, agent harness, model, or skill ecosystem used and may not directly apply to all settings.

    Source: B. Threats to Validity · p. 10

Abstract

from arXiv · show

Agent skills are the de facto mechanism for extending LLM agents with reusable guidance. A skill can shape the agent's task execution, including planning, tool use, problem-solving, and validation. Prior work reported mixed results of agent skills: some skills improve task success rates, while others have no effect, increase token use and execution time, and even reduce success rates. This paper presents a comprehensive analysis of skill-induced agent failures by attributing task failures and cost regressions to specific loaded skills. We introduce a differential analysis framework that attributes a failure or regression to a skill by comparing a target skill-guided run against a no-skill or semantically matched skill reference run that solves the same task, or solves it more cheaply. We instantiate this framework on SkillsBench and SWE-Skills-Bench, yielding 307 skill-induced failures, including 125 functional failures and 182 efficiency regressions. We also build SkillTriage, a taxonomy-guided attribution tool that normalizes paired cases, extracts differential evidence, and produces triage reports. Our major findings include: (1) Skill induced functional failures are rarely caused by obviously irrelevant skills; instead, seemingly relevant skills often make the agent incorrectly implement or omit task-required implementation elements. (2) Skill-induced efficiency regressions are not explained by prompt length alone. (3) The largest sources within Excessive Procedure are excessive verification and heavy implementation pipelines, contributing 67 and 30 cases, respectively. This shows that skills often turn validation checklists and construction recipes into mandatory work. Based on our findings, we propose research topics and tooling improvements for safer and more cost-aware skill reuse.

I. INTRODUCTION

Agent skills provide reusable procedural guidance but can mislead agents when their assumptions do not fit a task, causing functional failures or costly executions. This study uses paired skill and reference runs to analyze those harms, their causes, and automated attribution.

I. INTRODUCTION

  • A skill may be topically relevant yet operationally wrong, steering incompatible workflows, dependencies, or verification and causing task failure or higher cost.Because skills are reused, harmful guidance can bias future agent executions repeatedly.Source: §I · p. 1
  • Prior benchmarks report mixed skill effects, while existing evaluations offer limited contrastive evidence for attributing failures to a specific skill or explaining trajectory changes.Manual attribution also does not scale to continuously screening skills and updates.Source: §I · p. 1
  • 307 confirmed skill-induced failures comprise 125 functional failures and 182 high-confidence efficiency regressions.The study attributes failures by comparing a skill-guided run with a no-skill or semantically matched-skill reference on the same task.Source: §I · p. 2
  • Task-Implementation Fault accounts for 86 of 125 functional failures (68.8%), as relevant skills often cause required elements to be implemented incorrectly or omitted.Verifier-facing artifact and environment boundaries account for 24 and 13 cases, respectively.Source: §I · p. 2
  • Excessive Procedure accounts for 114 of 182 efficiency regressions (62.6%), led by excessive verification (67 cases) and heavy implementation pipelines (30 cases).The study reports that these regressions are not explained by prompt length alone.Source: §I · p. 2
  • SKILLTRIAGE normalizes paired cases, extracts differential evidence, and produces triage reports, matching exact human-assigned causes for 111/125 functional failures and 132/182 efficiency regressions.The study presents the tool as an automated aid for attributing skill-induced failures to root causes.Source: §I · p. 2

II. BACKGROUND

The paper treats skills as procedural packages that enter an agent’s context and can influence its actions. Its study design compares controlled runs that vary skill setup, measuring both correctness and execution cost.

II. BACKGROUND

  • An agent skill is a structured package of procedural knowledge loaded at inference time without changing model parameters.Skills may include applicability metadata, procedural guidance, supplementary resources, and validation guidance that become part of the agent’s effective task context.Source: §II · p. 2
  • Skill components can shape where artifacts are written, alter the execution environment, and add verification costs when their assumptions do not match the task.The study analyzes execution trajectories—the ordered model interactions, tool calls, and intermediate actions—rather than only final outcomes.Source: §II · pp. 2–3

A. Study Design

  • Paired executions hold the task, verifier, agent framework, model, repository or container state, and input data fixed while varying the skill setup.Correctness is measured by the task verifier or tests; cost is measured by token use and execution time.Source: A. Study Design · p. 3
  • A reference run uses no skill or a semantically matched skill, serving as a pseudo-oracle for whether the same task passes or costs less.The two failure patterns are target FAIL/reference PASS for functional failure and both runs PASS with substantially higher target cost for efficiency regression.Source: A. Study Design · p. 3
  • The four-stage methodology selects deterministic benchmark tasks, augments skill options, runs controlled evaluations, then labels and refines paired cases.Ambiguous, verifier-narrow, or duplicate cases are removed before root-cause analysis.Source: A. Study Design · p. 3

B. Study Subjects

The study uses SkillsBench and SWE-Skills-Bench, whose deterministic verifiers support reproducible pass/fail and cost comparisons. Their original skill settings provide a starting point but often lack alternative successful skills needed to localize failures to skill differences.

B. Study Subjects

  • SkillsBench covers 84 tasks across 11 domains, while SWE-Skills-Bench provides 490 repository-based software-engineering task instances; both use deterministic programmatic verifiers.SkillsBench includes no-skill, curated-skill, and self-generated-skill settings; SWE-Skills-Bench includes no-skill and curated-skill settings.Source: B. Study Subjects · p. 3
  • The original benchmark settings measure skill utility but offer a narrow comparison space for mechanism-level failure attribution.They can show that a skill helps, hurts, or adds cost, but often lack an alternative successful skill for the same task.Source: B. Study Subjects · pp. 3–4

C. Data Collection

The study expands benchmark comparisons with semantically matched public skills, then evaluates paired executions under controlled conditions. It labels and refines cases using correctness and cost evidence, producing a final dataset of confirmed skill-induced failures.

C. Data Collection

  • The public-skill search retains up to five candidates per curated skill with metadata similarity of at least 0.7, using cosine similarity over sentence embeddings.Candidates are retrieved from two public skill-sharing sites and are not forced into context; the agent may load or avoid them.Source: C. Data Collection · p. 4
  • The expanded comparison space grows from 826 to 20,664 potential paired comparisons, roughly 25×.Each task can have up to six skill variants, enabling both with/no-skill and cross-skill comparisons.Source: C. Data Collection · p. 4, Table I · p. 4
  • Each evaluation pair fixes the task, repository or container state, data, verifier, agent framework, and model, while recording skill use, trajectory, outcome, token use, and execution time.The study used OpenCode 1.15.1 and Claude Opus 4.6.Source: C. Data Collection · p. 4

D. Data Labeling

  • Efficiency regressions are identified from PASS/PASS pairs when both token and time ratios exceed 1.0 and at least one exceeds the primary threshold T = 2.0.The rule excludes token–time tradeoffs and labels cases as token-dominant, time-dominant, or joint regressions.Source: D. Data Labeling · pp. 4–5
  • From 20,664 potential pairs, the executed evaluations yield 665 labeled candidates that are refined by removing insufficient-evidence, likely verifier-induced, and duplicate cases.Remaining cases are manually inspected and root-cause labels are finalized through group consensus.Source: D. Data Labeling · p. 5, Table II · p. 5
  • The final analysis dataset contains 307 confirmed skill-induced failures: 125 functional failures and 182 high-confidence efficiency regressions.These cases follow candidate refinement and manual root-cause labeling.Source: D. Data Labeling · p. 5

IV. WHAT ARE THE ROOT CAUSES OF FUNCTIONAL FAILURES?

The taxonomy classifies confirmed functional failures by mechanism, with Task-Implementation Fault the largest category. Applicability and environment mismatches account for smaller but distinct failure groups.

IV. WHAT ARE THE ROOT CAUSES OF FUNCTIONAL FAILURES?

  • 86 of 125 confirmed failures (68.8%) are Task-Implementation Faults, followed by 24 Artifact Misplacement failures (19.2%).Table III summarizes the four-category classification; the supplied prose identifies these two largest categories.Source: §IV · p. 5, Table III · p. 6

A. Applicability Mismatch (APM)

  • 2 confirmed failures (1.6%) are Applicability Mismatches, where skill metadata signals applicability despite a mismatch with the concrete task scope.The distinction from artifact misplacement is that the error begins with the decision to apply the skill.Source: A. Applicability Mismatch (APM) · p. 5

B. Environment Mismatch (EM)

  • 13 confirmed failures (10.4%) are Environment Mismatches: 5 involve broken dependencies or runtimes, and 8 involve environment-state mismatches.The latter include changes to package resolution, working directory, installed versions, local installs, or tool state.Source: B. Environment Mismatch (EM) · p. 5
  • In an openpyxl case, the target validated against an external install while the reference patched the repository package, isolating environment state as the failure source.The comparison indicates the failure was not due to artifact content alone.Source: B. Environment Mismatch (EM) · p. 5

C. Task-Implementation Fault (TIF)

Task-Implementation Faults arise when on-topic skill guidance disrupts implementation of required task elements. Incorrect implementations and omissions account for most such failures.

C. Task-Implementation Fault (TIF)

  • 86 confirmed functional failures (68.8%) are Task-Implementation Faults, the largest category, where skill-induced implementation or artifact faults prevent satisfying required task elements.The skill may make the workflow too heavy, lead to an incorrect implementation, or leave a required element absent.Source: C. Task-Implementation Fault (TIF) · p. 6
  • Incorrect Required-Element Fill accounts for 46 cases (36.8%), while Required-Element Omission accounts for 36 (28.8%).The distinction is whether the required element is implemented incorrectly or left absent; Obstructive Workflow Guidance accounts for 4 cases (3.2%).Source: C. Task-Implementation Fault (TIF) · p. 6
  • In the RAG demo, the target exposed retrieval parameters through CLI flags but omitted the required model-parameter field, model_name.The partial skill example showed retrieval parameters but not model-parameter configuration.Source: C. Task-Implementation Fault (TIF) · p. 6
  • Only 2 of 125 functional failures (1.6%) are Applicability Mismatches; most involve task-implementation faults, often from treating reusable guidance as task-specific requirements.The authors recommend separating mandatory requirements from examples, defaults, reusable templates, and optional workflows.Source: C. Task-Implementation Fault (TIF) · p. 6

D. Artifact Misplacement (AM)

Artifact Misplacement covers cases where a plausible artifact is written or integrated somewhere other than the task-specified location.

D. Artifact Misplacement (AM)

  • 24 confirmed functional failures (19.2%) involve writing or integrating a required artifact at the wrong location, often by following repository or package conventions over the task path.The category concerns the specified output path or integration point, not whether the artifact itself is plausible.Source: D. Artifact Misplacement (AM) · p. 7
  • In a LangChain RAG task, the target wrote files under langchain_classic instead of the task-specified langchain directory, while the reference followed the specified paths.The decisive failure was artifact placement, not the plausibility of the generated components.Source: D. Artifact Misplacement (AM) · p. 7

V. WHAT ARE THE ROOT CAUSES OF EFFICIENCY REGRESSIONS?

The efficiency-regression taxonomy uses a conservative paired-run threshold and groups cases into context bloat, excessive procedure, and dependency resolution. Excessive Procedure is the largest category, while context costs chiefly come from always-loaded skill text.

V. WHAT ARE THE ROOT CAUSES OF EFFICIENCY REGRESSIONS?

  • 114 of 182 high-confidence efficiency regressions (62.6%) are Excessive Procedure, compared with 46 Context Bloat cases (25.3%).The primary threshold is T = 2.0: token use or execution time must more than double while the other metric also increases relative to the reference.Source: §V · p. 7

A. Context Bloat (CO)

  • 46 high-confidence regressions (25.3%) are Context Bloat; Skill-Body Context Bloat accounts for 43 cases (23.6%), versus 3 (1.6%) for Supplementary-Material Bloat.The skill body can make each model call costlier even when the task-solving trajectory remains similar.Source: A. Context Bloat (CO) · p. 7
  • 43 of 46 context-overhead cases stem from mandatory skill-body text, while supplementary-material bloat appears in only 3 cases.The authors suggest shortening always-loaded instructions and lazy-loading examples, templates, checklists, and background material.Source: A. Context Bloat (CO) · p. 7

B. Excessive Procedure (EP)

Excessive Procedure is the largest source of high-confidence efficiency regressions: skills add steps that increase exploration, implementation, or verification work. Dependency resolution is a separate cost category, while SKILLTRIAGE applies the taxonomy to paired cases for post-confirmation attribution.

B. Excessive Procedure (EP)

  • 114 of 182 high-confidence efficiency regressions (62.6%) are Excessive Procedure, where skills add exploration, implementation, retries, or verification steps.Excessive Verification contributes 67 cases; Heavy Implementation Pipeline contributes 30.Source: B. Excessive Procedure (EP) · pp. 7–8
  • Excessive Procedure differs from Context Bloat because it changes the task-solving trajectory, whereas Context Bloat raises per-call cost while leaving the sequence of steps similar.When both occur, the subcategory follows the larger observed cost driver.Source: B. Excessive Procedure (EP) · p. 8

C. Dependency Resolution (DO)

  • Dependency Resolution accounts for 22 high-confidence regressions (12.1%) in which a passing task requires setup or repair of a fragile or incompatible runtime dependency.If the dependency problem remains unresolved and causes verifier failure, it is classified as Broken Dependency or Runtime in the functional-failure taxonomy.Source: C. Dependency Resolution (DO) · p. 8

VI. AUTOMATED ATTRIBUTION OF SKILL-INDUCED AGENT FAILURES

  • SKILLTRIAGE takes an already-confirmed target/reference pair and predicts its category and subcategory, returning contrastive evidence for the attribution.It supports triage and is evaluated against manual labels; it does not create the taxonomy or replace manual validation.Source: §VI · p. 8

A. Attribution Tool

SKILLTRIAGE operationalizes the manual taxonomy by comparing normalized target and reference runs and extracting differential evidence for attribution. Evaluated against manual labels, it matches exact subcategories in most cases, with residual errors concentrated near taxonomy boundaries.

A. Attribution Tool

  • SKILLTRIAGE extracts differential signals from paired target/reference runs, then selects the root cause that best explains both the outcome and trajectory divergence.Functional signals inspect skill scope, environment, implementation, and artifact location; efficiency signals track phase- and action-level cost changes.Source: A. Attribution Tool · pp. 8–9
  • The tool returns a category, subcategory, natural-language rationale, cited skill or trajectory evidence, and a repair suggestion.For example, a task-path mismatch is attributed to Wrong Artifact Location rather than an implementation-content subcategory.Source: A. Attribution Tool · p. 9

B. Evaluation Setup

  • For evaluation, the authors ran GPT-5.5 three times on each paired case and used a 2-of-3 majority vote against manually assigned labels.The evaluation covered 125 confirmed functional failures and 182 high-confidence PASS/PASS efficiency regressions at T = 2.0; benchmark tasks were not rerun.Source: B. Evaluation Setup · p. 9

C. Attribution Results

  • SKILLTRIAGE matched manual functional-failure labels in 117/125 cases (93.6%) at category level and 111/125 (88.8%) at exact subcategory level.Within Task-Implementation Fault cases, exact agreement was 76/86 (88.4%).Source: C. Attribution Results · p. 9, Table V · p. 10
  • For efficiency regressions, SKILLTRIAGE matched manual labels in 145/182 cases (79.7%) at category level and 132/182 (72.5%) at exact subcategory level.Within Excessive Procedure cases, exact agreement was 78/114 (68.4%).Source: C. Attribution Results · pp. 9–10, Table VI · p. 10
  • The remaining exact-subcategory errors cluster near neighboring taxonomy boundaries, where the same differential evidence can support multiple labels.Efficiency cases can contain overlapping cost surfaces, such as dependency repair within verification or build commands that could count as implementation or verification.Source: C. Attribution Results · p. 10

VII. DISCUSSION

The discussion proposes compatibility checks and cost-aware controls for skill reuse, while noting that the study’s findings may depend on its benchmarks and experimental setting.

A. Future Research Directions

  • Future platforms could check task requirements against candidate skills for conflicts in fields, APIs, paths, output formats, domain rules, or environment assumptions.The authors also propose estimating both context cost and induced trajectory work, including exploration, implementation, verification, and dependency repair.Source: A. Future Research Directions · p. 10
  • Agents could adapt verification scope and implementation-pipeline depth to task uncertainty, repository size, change risk, and remaining budget.This targets excessive verification and heavy implementation pipelines without making careful skill-guided workflows universally mandatory.Source: A. Future Research Directions · p. 10

B. Threats to Validity

  • Manual root-cause analysis involves judgment about verifier scope, ordinary run variance, and target/reference divergence; ambiguous and verifier-narrow cases were excluded.The authors reached group consensus before finalizing taxonomy labels.Source: B. Threats to Validity · p. 10
  • Findings may depend on the task distribution, agent harness, model, or skill ecosystem and may not generalize to all settings.The study covers SkillsBench and SWE-Skills-Bench tasks across multiple domains, but avoids conclusions specific to only those benchmarks.Source: B. Threats to Validity · p. 10

VIII. RELATED WORK

Prior work examines skill utility, context effects, and agent failures, while this paper focuses on attributing failures and cost regressions to loaded skills. The conclusion frames skill loading as a configuration decision requiring validation and cost awareness.

VIII. RELATED WORK

  • SkillsBench and SWE-Skills-Bench report mixed utility, while this study asks which skill contents and trajectory changes underlie failures and efficiency regressions.Related work also studies context effects and agent failures, but the paper’s attribution target is the skill under contrastive comparisons.Source: §VIII · pp. 10–11

IX. CONCLUSION

  • The paper concludes that 307 confirmed cases comprise 125 functional failures and 182 high-confidence efficiency regressions, with regressions driven mainly by Excessive Procedure rather than prompt length alone.It proposes treating skill loading as a costed, validated configuration decision and presents SKILLTRIAGE for taxonomy-guided attribution.Source: §IX · p. 11

Related papers

Loading 2608.11888v1…