Source-linked AI summary
Demystifying Agent Skills: Why They Work-Until They Don't
Zhiyuan Jiang, Fangrui Huang, Hanwen Xing, Xander Wu, Yipeng Gao, Rui Cao, Mengdi Wang, Shilong Liu, Yijiang Li
TL;DR
Existing evaluations show whether skills improve task success but not when they help, why they work, or where they fail. This paper uses controlled experiments and contrastive trajectory analysis to identify skill-use mechanisms, finding that skills are most effective as procedural anchors that stabilize execution.
Problem
Existing evaluations measure aggregate task success but provide limited evidence about how skills change agent behavior, when they help, and why they fail.
Method
The study compares matched executions with raw, workflow-memory, and skill injections while analyzing paired trajectories across the skill-use pipeline.
Results
Skills are most useful when they convert noisy prior experience into procedural guidance, with procedural anchoring accounting for 65.7% of cases versus 4.5% for factual knowledge injection.
Takeaways & Limitations
Reliable skill-based agents require lifecycle-aware evaluation and design spanning how procedural knowledge is represented, retrieved, invoked, and adapted.
Takeaways & Limitations
The evaluation covers terminal- and tool-using benchmarks, limited agent–model configurations, and a taxonomy sampled from approximately 3% of normalized records.
Abstract
from arXiv · showhide
Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: \emph{\textbf{When do skills help, why do they work, and where do they fail?}} Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7\% of skill cases, versus 4.5\% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6\% to 3.3\%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents.
1 Introduction
This study moves beyond aggregate task success to explain when skills help, why they work, and where they fail. It finds that skills primarily convert noisy experience into procedural anchors, but remain dependent on retrieval, contextual compatibility, and adaptation.
- Motivation and methodology: The study evaluates skill utility through matched executions and paired trajectory analysis rather than treating skills as a black box.It examines how behavior changes before and after skill access across the skill-use pipeline.
- Core findings: 65.7% of skill cases reflect procedural anchoring, whereas explicit factual knowledge injection accounts for 4.5%.Skills primarily help agents follow more reliable setup steps, tool sequences, implementation routines, and verification checks.
- Core findings: Skills reduce execution-layer failures, while workflow memory can preserve irrelevant exploration, failed branches, and verbose process noise that increase timeout and drift.Observed failure reductions include environment setup errors, output-format mismatches, service-lifecycle failures, and shell-command corruption.
- Limitations and lifecycle: Skills work when distilled experience is retrieved and applied as a compatible procedural anchor, but fail when guidance is noisy, over-specific, mismatched, weakly invoked, or followed without adaptation.Hard distractors impair explicit selection, yet correct retrieval does not guarantee success and retrieval misses are not the only failure source.
- Motivation and methodology: 8,135 trial records and 240 sampled trajectories yield 238 valid labels organized into three categories and twelve skill-use modes.The taxonomy supports systematic analysis of skill effects and failures.
2 Related Works
Related work has progressed from episodic and structured memory toward reusable procedural workflows and explicit skills, including their activation and evolution. Existing benchmarks and evaluations largely emphasize aggregate task success, motivating analysis of how memory decisions shape skill outcomes.
- Memory and procedural knowledge: Memory systems retrieve episodic experiences, reflections, interaction histories, structured external memory, or past experiences to support agents across tasks.This line of work focuses on remembering and reusing prior interactions for planning and decision-making.
- Memory and procedural knowledge: Workflow memories summarize trajectories into action-level procedures, while skill systems store reusable procedures as executable routines, scripts, directories, or hierarchical knowledge bases.Skills make procedural knowledge explicit as first-class artifacts rather than leaving it embedded in broader experience memory.
- Skill lifecycle and evaluation: Recent studies examine skill activation, selection, discovery, revision, and evolution through experience, but skills are still often evaluated mainly through aggregate task success.This evaluation emphasis obscures how skills differ from broader procedural memory and why they help or fail in specific execution settings.
- Skill lifecycle and evaluation: This work addresses the gap by comparing skills with procedural memory and attributing effects to representation, retrieval, invocation, transfer, and abstraction stages.The analysis connects skill behavior to specific stages rather than treating final success as the sole outcome.
- Benchmarks: Existing benchmarks cover tool use, computer use, software, terminal, and explicit skill settings, but primarily report final task success rather than specific memory decisions.Terminal-Bench offers long-horizon command-line tasks with isolated environments and test-based verification, while SkillsBench and SWE-Skills-Bench explicitly evaluate skills.
3 Study Design
The study uses controlled experiments and trajectory analyses to determine how representation, outcome signals, framework transfer, and retrieval shape skill use. It holds underlying experience fixed where needed and separately measures procedural reuse, portability, retrieval, and downstream execution.
- Procedural experience representation: The study compares Raw, Workflow Memory, and Skill conditions built from identical prior trajectories to isolate how procedural representation changes experience reuse.Balanced success/failure trajectory pools and fixed-budget mixtures vary source evidence from success-only to failure-only.
- Outcome signals: Standard and no-hint Skill variants remove success/failure annotations while preserving trajectories and execution, isolating procedural content from explicit outcome signals.The no-hint setting tests whether outcome information contributes beyond the reusable procedures contained in prior trajectories.
- Cross-framework transfer: Cross-framework transfer fixes source experience in Codex and evaluates it in Gemini CLI, testing portability across prompting styles, tool interfaces, and execution behavior.RQ3 compares skills and workflow memories after changing the target agent framework.
- Skill retrieval and downstream execution: RQ4 evaluates skill retrieval and downstream execution with complete candidate pools, measuring selection quality, execution-time access, and final verifier outcomes.The experiments separately assess embedding retrieval, explicit agent selection, and full-pool real execution.
- Skill retrieval and downstream execution: Candidate pools contain ground-truth skills plus random, semantically similar, or dissimilar distractors, with pool sizes ranging from 5 to 100.These measurements are treated as independent rather than sequential stages.
4 Skill-Use Mechanisms: A Contrastive Taxonomy
The paper builds a contrastive trajectory-analysis pipeline and stable taxonomy to explain how skills alter agent execution beyond aggregate success rates. Across paired task settings, skills increase procedural anchoring and reduce execution failures, while also introducing more invocation or boundary failures than raw execution.
- Contrastive analysis: 8,135 trial records were normalized into a shared manifest for contrastive analysis of raw, workflow-memory, and skill execution.The records include task identity, execution arm, verifier outcome, injected artifact, and available trajectory transcripts.
- Taxonomy construction: 95.8% exact agreement and Cohen’s κ = 0.952 show that human and LLM taxonomy assignments are highly consistent.Human validation independently mapped 238 raw labels to the 12 canonical modes, supporting taxonomy stability beyond a single LLM’s judgments.
- Contrastive analysis: 528 paired triples compared raw execution, workflow-memory injection, and skill injection across SkillsBench, Terminal-Bench 2.0, and Terminal-Bench-Pro.The triples yielded 1,584 arm-level mode assignments for LLM-based taxonomy and pairwise-change judgments.
- Taxonomy construction: The taxonomy organizes trajectories into three high-level Skill-use Categories containing 12 fine-grained modes, including procedural anchoring, execution-layer failures, and invocation or boundary failures.SC1 captures successful procedural anchoring, SC2 captures execution-layer and verification failures, and SC3 captures guidance that is misused, over-applied, ignored, or externally constrained.
- Observed skill-use modes: 326/528 skill-arm SC1 assignments versus 294/528 for workflow memory show that skills shift more trajectories into successful procedural anchoring.Skills also reduce SC2 execution-layer failures to 124/528 versus 197/528 for raw and 176/528 for workflow memory.
- Observed skill-use modes: 78/528 skill-arm SC3 invocation or boundary failures versus 19/528 for raw execution show that skills can increase failure at invocation or task boundaries.The category-level comparison decomposes skill effects into procedural gains, fewer execution-layer failures, and additional invocation or boundary failures.
5 Findings
Skills outperform workflow memory by distilling trajectories into procedural anchors that stabilize execution, especially against operational fragility. Their benefits remain bounded by misapplication, deeper reasoning failures, outcome-label dependence, and retrieval difficulty as candidate pools expand.
- Aggregate effectiveness: +6.06 percentage points: skills improve over workflow memory, reaching 61.9% oracle-status success versus 59.1% for raw execution and 55.9% for workflow memory.Because both baselines derive from the same source trajectories, the gain reflects representation rather than merely additional prior experience.
- Aggregate effectiveness: 61.6% of skill-arm cases are skill_guided_success, versus 54.5% workflow_guided_success, indicating that cleaner procedural compression outperforms trace-level workflow memory.Workflow memory remains useful because traces contain reusable commands, parameter choices, and debugging evidence, but also preserve incidental process and failed attempts.
- Operational robustness: 23.5%: SC2 execution-layer and verification failures in skill arms, compared with 37.3% for raw execution and 33.3% for workflow memory.Environment_infrastructure_failure falls from 5.3% in raw execution to 1.7% with workflow memory and 0.2% with skills.
- Failure boundaries: 7.4%: algorithmic_logic_error persists with skills, alongside 11.7% static_verification_without_runtime, showing that procedural guidance does not repair wrong algorithms or enforce runtime validation.The corresponding raw and workflow-memory rates are 8.3% and 11.0% for algorithmic_logic_error, and 12.5% for static_verification_without_runtime.
- Failure boundaries: 10.0%: skill_guidance_misapplied_or_ignored in skill-arm cases, versus 0.8% in raw execution and 0.4% in workflow memory, while workflow timeout_budget_exhaustion reaches 10.6%.These contrasting failures show that skills introduce adaptation and invocation risks, whereas direct traces can overwhelm agents with procedural residue.
- Retrieval and outcome dependence: 29.6% to 3.3%: actual-use precision falls as pools grow from 5 to 100, while downstream success changes only from 36.4% to 39.3%.Offline top-1 embedding precision also decreases from 88.3% to 76.9%, and failed trajectories increase the value of outcome-aware skill creation, reaching 0.7462 versus 0.4000 without outcome hints.
6 Conclusion
The paper finds that skills work primarily as procedural anchors, while failing under retrieval errors, contextual mismatch, rigid execution, or tasks requiring deeper reformulation and runtime validation. Reliable self-evolving agents therefore require lifecycle improvements in representing, retrieving, and leveraging procedural knowledge.
- 6 Conclusion: Skills are most effective as procedural anchors, shifting evaluation beyond aggregate success rates toward when they help, why they work, and where they fail.The study combines controlled experiments with contrastive trajectory analysis to examine skill behavior.
- 6 Conclusion: Skills fail when retrieved incorrectly, invoked in the wrong context, followed too rigidly, or applied to tasks requiring deeper reformulation and runtime validation.These limitations identify conditions under which skill use does not reliably support execution.
- 6 Conclusion: Skill use is a lifecycle problem requiring better representation, retrieval, and leveraging of procedural knowledge—not merely the generation of more skills.This framing supports more principled evaluation and design of future skill-based agent systems.
7 Limitations
The evaluation is limited to terminal- and tool-using benchmarks and a small set of agent–model configurations, so its findings may not generalize across the full range of agentic behavior or other systems.
- Evaluation scope: The benchmarks emphasize multi-step execution, debugging, and verification rather than long-horizon web interaction or open-ended collaboration.This leaves broader forms of agentic behavior outside the evaluation scope.
- Generalizability: The study evaluates a limited number of agent–model configurations, constraining generalization to other scaffolds, model families, or model versions.The authors plan to extend the study to broader settings in future work.
A Implementation Details
The experiments use controlled comparisons to isolate how representation, outcome annotation, cross-framework transfer, and retrieval difficulty affect results. Conditions generally share the same tasks, interface, framework, model, harness, and trial budget.
- A Implementation Details: Controlled comparisons isolate representation, outcome annotation, cross-framework transfer, and retrieval difficulty.These experiments address four research questions described in the main text.
- A Implementation Details: Unless otherwise stated, experiments use the same target tasks, benchmark interface, agent framework, model, execution harness, and trial budget across conditions.This standardization supports comparisons focused on the intended experimental factor.
- A Implementation Details: The implementation details cover four research questions through experiments designed to isolate individual sources of variation.The stated factors are representation, outcome annotation, cross-framework transfer, and retrieval difficulty.
A.1 Model–Framework Pairings and Benchmarks
The evaluation spans realistic agent-framework deployments and controlled retrieval settings, using multiple model–framework pairings across Terminal-Bench and SkillsBench. Experiments combine controlled benchmark subsets with standardized downstream execution and retrieval-isolation protocols.
- Model–framework pairings: RQ1–RQ3 use Codex + GPT-5.3-Codex and Gemini CLI + Gemini-3.1-Pro-Preview, while RQ4 adds Gemini CLI + Gemini-3.1-Pro-Preview and Codex + GPT-5.4.RQ3 transfers workflow memories and skills from the primary Codex setting to Gemini CLI; RQ4 uses Qwen3-Embedding-0.6B for embedding retrieval.
- Benchmarks: The evaluation suite combines Terminal-Bench and SkillsBench, with RQ1–RQ3 drawing controlled subsets from three public benchmark splits.Terminal-Bench Pro has a 200-task public split from a 400-task, 8-domain benchmark; Terminal-Bench 2.0 has 89 tasks; SkillsBench has 86 tasks across 11 domains.
- Evaluation protocols: Downstream execution follows Harbor’s standard workflow with n = 5 unique trials per task and a parallelism of 20 unless otherwise noted.Retrieval-isolation arms use one query per task–pool setting because they perform no benchmark execution.
- Evaluation protocols: Skills are placed in the agent’s execution environment as reusable procedural resources under a standard skill-use protocol.This setup supports evaluation in both realistic agent-framework deployments and controlled retrieval settings.
A.2 Dive into Skill-Use Mechanisms: Trajectory Labeling and Comparative Analysis … A.8 Lightweight Compact Procedural Baselines
The paper uses paired trajectory analysis to explain how prior-experience representations alter execution, identifying procedural anchoring alongside retrieval and transfer bottlenecks. Across controlled comparisons, it tests representation, outcome annotation, cross-framework portability, retrieval, and compact procedural-text baselines.
- A.2 Dive into Skill-Use Mechanisms: Trajectory Labeling and Comparative Analysis: 238 valid unique labels induced a canonical taxonomy of 12 mixed success/failure modes, distinguishing skill guidance, workflow guidance, and autonomous execution.The taxonomy was built from open-ended trajectory labels before being applied to 528 paired triples.
- A.2 Dive into Skill-Use Mechanisms: Trajectory Labeling and Comparative Analysis: Paired triples compare Raw, Workflow Memory, and Skill executions for the same benchmark tasks, isolating which failure modes prior-experience representations fix or introduce.The central design compares matched trajectories rather than measuring failure frequencies independently.
- A.3 Paired-Trajectory Example: The analysis aggregates paired success deltas, mode frequencies, fixed and introduced modes, mechanism distributions, and benchmark- and setting-level trends without additional LLM calls.The paired dataset contains 528 triples, while representative-trial selection and incomplete skill-arm token metadata limit interpretation.
- A.4 RQ1: Representation of Prior Experience: RQ1 holds tasks, models, interfaces, scaffolds, and trial budgets constant while comparing Raw, Workflow Memory, and Skill representations of the same underlying trajectories.Workflow Memory preserves cleaned procedural flow, whereas Skill distills those workflows into standardized SKILL.md artifacts.
- A.5 RQ2: Outcome Annotation and No-Hint Ablation: RQ2 removes explicit success/failure annotations during skill construction while preserving workflow content, task pools, experience budgets, and standardized skill format.Matched reruns compare standard and no-hint Skill using downstream success rate and token/context cost.
- A.6 RQ3: Cross-Framework Transfer: RQ3 evaluates Codex-generated Workflow Memory and Skill artifacts with Gemini CLI, treating downstream task success under the new framework as portability.The framework shift changes prompting style, tool interface, and execution loop while keeping target tasks and source experience fixed.
- A.7 RQ4: Skill Retrieval and Downstream Execution: RQ4 separates offline semantic retrieval, deliberate agent selection, and real execution, varying candidate pools from 5 to 100 under random, similar, and dissimilar distractors.Native task–skill annotations define ground truth, and execution parses inspected or invoked skills alongside precision, recall, F1, and benchmark success.
- A.8 Lightweight Compact Procedural Baselines: Compact procedural baselines test whether skill benefits arise from short text alone using a three-to-five-step short plan and a test-first validation template.Both plain-text baselines run on 26 selected Terminal-Bench-2 tasks with five trials per task and are compared against Raw, Workflow Memory, and Skill.
A.9 Matched Token-Cost Analysis · B Prompts
Matched analysis on 83 tasks shows a trade-off: Workflow Memory is most token-efficient, while Skill achieves the highest success rate with stronger execution performance.
- A.9 Matched Token-Cost Analysis: 83 tasks form the matched intersection for comparing Raw, Workflow Memory, and Skill token usage.Only tasks with usable token metadata across all three representations were included.
- A.9 Matched Token-Cost Analysis: Success and token usage were first averaged within each task and representation, then averaged across tasks.This prevents tasks with more completed trials from receiving disproportionate weight.
- A.9 Matched Token-Cost Analysis: The same-task comparison keeps the task mix fixed when evaluating representation choices.This controls comparisons across Raw, Workflow Memory, and Skill.
- A.9 Matched Token-Cost Analysis: Workflow Memory is the most token-efficient representation, reducing both input and output tokens relative to Raw trajectories.Its efficiency advantage is measured against the Raw representation.
- A.9 Matched Token-Cost Analysis: Skill is not uniformly cheaper than Workflow Memory, creating an effectiveness–efficiency trade-off.Skill can require additional context relative to Workflow Memory.
- A.9 Matched Token-Cost Analysis: Skill achieves the highest success rate, improving over Raw while also reducing token usage.Its advantage combines stronger outcomes with lower usage than Raw trajectories.
- A.9 Matched Token-Cost Analysis: Relative to Workflow Memory, Skill trades additional context for stronger execution performance.The paper therefore characterizes Skill as more effective and Workflow Memory as more token-efficient.
B.1 skill-creator.md (Experiment 1) · B.2 skill-creator-no-hint.md (Experiment 2) · B.3 Prompts for Trajectory Labeling and Comparative Analysis
The appendices define trace-to-skill generation prompts, a no-hint variant, and structured procedures for labeling, consolidating, and comparing agent trajectories. Together, they operationalize reusable procedural skills, evidence-grounded failure analysis, canonical mode taxonomies, and mechanism-level skill comparisons.
- B.1 skill-creator.md (Experiment 1): Skill generation extracts repeatable processes, ordered steps, tool usage, cross-trace patterns, and failure modes from execution traces.The resulting file is intended to capture a generalizable process rather than a task-specific instance.
- B.1 skill-creator.md (Experiment 1): The generated skill uses sections for triggering conditions, preconditions, concrete steps, failure modes, recovery actions, and verification.The recovery procedure stops to inspect output, maps errors to documented failure modes, applies fixes, and reruns verification.
- B.1 skill-creator.md (Experiment 1) · B.2 skill-creator-no-hint.md (Experiment 2): Both creator variants require exactly one reusable, concrete, generalized skill and prohibit task-specific paths or data by using placeholders.When traces differ, the prompts direct the generator to select the most reliable approach.
- B.2 skill-creator-no-hint.md (Experiment 2): The no-hint creator infers failure signals and mitigations only from observable trace evidence, without assuming whether a run succeeded or failed.Examples of evidence include command outputs, error text, exit status, and retries.
- B.3 Prompts for Trajectory Labeling and Comparative Analysis: Trajectory labeling compares raw, workflow-memory, and curated-skill conditions while requiring concrete explanations supported by tool calls or output snippets.The JSON schema records a primary mode, secondary factors, evidence spans, skill-effect judgment, and capability-versus-knowledge classification.
- B.3 Prompts for Trajectory Labeling and Comparative Analysis: Canonical mode assignment covers environment failures, API misuse, debugging loops, verification mismatches, timeouts, skill-specific effects, workflow patterns, and successful execution.Every input receives exactly one mode, with specific procedural labels preferred over generic catch-alls.
- B.3 Prompts for Trajectory Labeling and Comparative Analysis: Taxonomy consolidation maps overlapping batch-level modes into 9-14 unified canonical modes while preserving the source-mode mapping.Every input batch mode must appear as a mapping key, and every mapped value must be defined in the unified taxonomy.
- B.3 Prompts for Trajectory Labeling and Comparative Analysis: Comparative analysis records per-arm modes, evidence quotes, outcome deltas, fixed or introduced modes, and whether the skill acted as knowledge injection, procedural anchor, warning, none, or counterproductive.Missing arms are marked non-comparable, and mechanism labels distinguish unused skills from skills that worsen performance relative to raw.
C Complete Results for Skill Retrieval and Outcome Annotation Ablation
This section reports complete retrieval results across embedding ranking, explicit agent selection, and real execution, alongside an outcome-annotation ablation. The experiments distinguish retrieval quality from downstream execution success and compare trajectory construction with and without source-outcome hints.
- Skill Retrieval: The retrieval evaluation spans embedding ranking, explicit agent selection, and parsed actual skill use across varying agent–model, distractor, and pool-size conditions.Arm 1 uses task–skill-description similarity from Qwen3-Embedding-0.6B; Arms 2 and 3 report retrieval and execution metrics, respectively.
- Skill Retrieval: Arm 1 reports ranking metrics, while Arms 2 and 3 report precision, recall, and F1, with downstream success included for real execution.Top-5 is omitted for k = 5 because it covers the full candidate pool, and dashes mark excluded entries.
- Outcome Annotation: The outcome-annotation ablation compares skills built with source-trajectory outcomes exposed against skills built with outcomes withheld.Terminal-Bench-Pro uses 130 trials per condition, counting missing or infrastructure-error trials as failures.