PaperlayerView exact revision on arXiv

Source-linked AI summary

SoK: Agentic Skills -- Beyond Tool Use in LLM Agents

Yanna Jiang, Delong Li, Haiyu Deng, Baihe Ma, Xu Wang, Qin Wang, Guangsheng Yu

arXiv:2602.20867v1cs.CRcs.AIcs.CEcs.ET

TL;DR

LLM agents need reusable procedural capabilities because otherwise they repeatedly reconstruct strategies and lose experience across context windows. This SoK formalizes agentic skills, maps their lifecycle and design space, and examines execution, security, governance, and evaluation; its benchmark synthesis reports benefits from curated skills but degradation from self-generated ones. The paper identifies reliability, verification, and governance as central boundaries for skill-based agents.

  • Problem

    LLM agents often re-derive execution strategies for recurring tasks because procedural knowledge disappears at the end of each context window.

  • Method

    The paper formalizes skills and systematizes their lifecycle, representations, scopes, design patterns, execution models, security, governance, and evaluation.

  • Results

    Curated skills improve agent success rates, while self-generated skills can degrade performance; the evaluation framework and case studies document this contrast.

  • Takeaways & Limitations

    Reliable skill-based agents require attention to skill quality, verification, governance, and the trade-off between scalable autonomous generation and dependable human curation.

  • Takeaways & Limitations

    The systematization analyzes 24 systems from 65 retained papers, and its bottom-up seven-pattern taxonomy lacks external expert-survey or formal validation.

Abstract

from arXiv · show

Agentic systems increasingly rely on reusable procedural capabilities, \textit{a.k.a., agentic skills}, to execute long-horizon workflows reliably. These capabilities are callable modules that package procedural knowledge with explicit applicability conditions, execution policies, termination criteria, and reusable interfaces. Unlike one-off plans or atomic tool calls, skills operate (and often do well) across tasks. This paper maps the skill layer across the full lifecycle (discovery, practice, distillation, storage, composition, evaluation, and update) and introduces two complementary taxonomies. The first is a system-level set of \textbf{seven design patterns} capturing how skills are packaged and executed in practice, from metadata-driven progressive disclosure and executable code skills to self-evolving libraries and marketplace distribution. The second is an orthogonal \textbf{representation $\times$ scope} taxonomy describing what skills \emph{are} (natural language, code, policy, hybrid) and what environments they operate over (web, OS, software engineering, robotics). We analyze the security and governance implications of skill-based agents, covering supply-chain risks, prompt injection via skill payloads, and trust-tiered execution, grounded by a case study of the ClawHavoc campaign in which nearly 1{,}200 malicious skills infiltrated a major agent marketplace, exfiltrating API keys, cryptocurrency wallets, and browser credentials at scale. We further survey deterministic evaluation approaches, anchored by recent benchmark evidence that curated skills can substantially improve agent success rates while self-generated skills may degrade them. We conclude with open challenges toward robust, verifiable, and certifiable skills for real-world autonomous agents.

1. Introduction

LLM agents increasingly handle multi-step autonomous workflows, but they often re-derive strategies instead of retaining procedural knowledge. This SoK introduces agentic skills as the organizing abstraction and maps their lifecycle, design patterns, representations, security, governance, and evaluation.

  • Motivation: LLM agents perform increasingly complex workflows, yet procedural knowledge gained during one task disappears after the context window ends.The paper contrasts this inefficiency with repeated debugging tasks that are approached as novel.
  • Agentic skills: Agentic skills are reusable, callable modules that package procedural knowledge for achieving classes of goals under recurring conditions.They differ from tools, plans, and episodic memories by being executable, reusable, and governable.
  • Agentic skills: Skills carry applicability conditions, termination criteria, and callable interfaces, making procedural knowledge a first-class operational unit.These properties distinguish skills from atomic tools, one-time plans, and stored observations.
  • Research gap: Existing surveys address LLM agents, tool use, or multi-agent coordination, but do not trace the skill lifecycle from acquisition through governance.The paper positions its skill-centric lens as filling this gap.
  • Contributions: The SoK contributes a unified skill definition, lifecycle model, seven-pattern design taxonomy, orthogonal representation × scope taxonomy, security analysis, and evaluation framework.The contributions cover how skills are defined, packaged, distributed, governed, and assessed.

2. What Is an Agentic Skill?

The paper defines an agentic skill as a reusable, callable module whose applicability, policy, termination, and interface make procedural knowledge executable and governable. This abstraction distinguishes skills from tools, plans, memory, and prompt templates while framing them as procedural memory for agents.

  • Formal definition: An agentic skill is formalized as S = (C, π, T, R), a four-component schema for reusable procedural capabilities.The paper presents the tuple as representation-agnostic and applicable to prompts, code, learned policies, or hybrids.
  • Formal definition: C determines applicability from observations and goals, while π maps observations and history to actions or skill invocations.The policy can support hierarchical composition by selecting another skill from the library.
  • Formal definition: T specifies when execution ends relative to the current goal, and R exposes the skill’s name, parameters, and return type for invocation.The interface supports programmatic use by agents, other skills, or external orchestrators.
  • Formal definition: Removing any tuple component eliminates self-selection, composability, programmatic invocation, or executability, respectively.The four components therefore capture complementary properties rather than redundant metadata.
  • Skills versus related abstractions: Skills differ from tools, plans, memory, and prompt templates because they combine executable policies, reuse across sessions, applicability logic, termination criteria, and callable interfaces.The paper compares these abstractions across reuse, execution, verification, composability, and governance surfaces.
  • Skills as procedural memory: Skills function as procedural memory by compressing recurring multi-step procedures into reusable modules that reduce repeated first-principles reasoning.The paper connects this role to procedural memory and reports that curated skills raise average pass rates by 16.2 percentage points, whereas self-generated skills reduce them by 1.3 pp.

3. Methodology

The paper uses a structured literature review and iterative taxonomy-development process to analyze agentic skills. It searches six databases, retains 65 papers from approximately 180 candidates, and validates categories against the analyzed corpus.

  • Literature search and selection: The review searched six databases using skill- and agent-related keywords plus forward and backward citation tracing from seed papers.The databases were Google Scholar, Semantic Scholar, DBLP, ACM Digital Library, IEEE Xplore, and arXiv.
  • Literature search and selection: The search covered January 2020 through February 2025 for LLM-agent systems, with SkillsBench included as a concurrent exception and foundational works included regardless of date.The stated exception was included because of its direct relevance to skill evaluation.
  • Literature search and selection: Approximately 180 candidate papers were screened, yielding 65 papers for detailed analysis, including 24 systems mapped in depth.The corpus spans eight benchmark environments, seven design patterns, and five representation categories.
  • Taxonomy development: The taxonomies began with a feature matrix recording representation, acquisition, execution, storage, and governance features for each analyzed system.Recurring clusters in the matrix suggested the seven design patterns and five representation categories.
  • Taxonomy development: Three revision cycles tested the candidate categories against the full corpus until deeply analyzed systems could be classified without forcing.The representation × scope taxonomy was developed orthogonally from the environments addressed by the systems.
  • Taxonomy development: The design patterns are non-exclusive because deployed systems often combine multiple packaging, loading, and implementation patterns.The paper treats this composability as a feature rather than a deficiency of the framework.

4. Skill Lifecycle Model

The lifecycle model treats skills as evolving components that move from discovery and refinement through distillation, storage, retrieval, composition, execution, evaluation, and update. Feedback loops connect stages when failures, retrieval problems, or underperformance require revision or new skill discovery.

  • Lifecycle model: The lifecycle comprises seven stages: discovery, practice/refinement, distillation, storage, retrieval/composition, execution, and evaluation/update.It traces a skill from initial formation to eventual retirement rather than treating it as a static artifact.
  • Discovery: Discovery identifies recurring task patterns or bottlenecks that justify encapsulating behavior into reusable skills, using curricula, decomposition, or demonstrations in representative systems.Voyager, DEPS, and AppAgent illustrate these discovery routes.
  • Practice/Refinement and Distillation: Practice/refinement improves candidate reliability through repeated execution, reflection, and external feedback, while distillation extracts stable procedures into compact representations.Practice changes reliability through iteration; distillation changes representation, such as from trajectories to code or model weights.
  • Storage and Retrieval/Composition: Storage persists skills with indexing, versioning, and metadata, while retrieval/composition selects compatible skills for higher-level workflows.Systems balance retrieval precision against recall and may incorporate task similarity, environmental state, interfaces, and dependencies.
  • Execution: Execution enacts a skill’s policy under representation-specific mechanisms, sandboxing, permissions, and resource constraints that bound side effects.Natural-language skills enter context, code skills run in sandboxes, and policy skills operate through learned parameters; CodeAct uses executable Python for expressiveness and verifiability.
  • Evaluation/Update: Evaluation/update monitors deployed performance for drift or failure and revises, replaces, or retires skills as requirements change.SkillsBench pairs 86 tasks with deterministic verifiers across 7,308 agent trajectories for reproducible evaluation.
  • Feedback loops: The lifecycle is non-linear: feedback links evaluation to practice, retrieval to storage, and execution failures to discovery.These loops support refinement, improved indexing, and creation of new skills when runtime behavior reveals unmet needs.

5. Design Patterns and Taxonomy

The paper classifies agentic skills using complementary system-level design patterns and an orthogonal representation × scope taxonomy. The design patterns span increasing autonomy and cover distinct choices in packaging, loading, and execution.

  • Seven design patterns describe how infrastructure packages, loads, and executes agentic skills, while the representation × scope taxonomy describes what skills are and where they operate.
  • Pattern-1: Metadata-Driven Disclosure: Pattern-1 uses compact metadata for discovery and loads full instructions only after selection, conserving the finite context window.This enables an agent to know about many skills while loading only activated specifications.
  • Pattern-1: Metadata-Driven Disclosure: Metadata-driven disclosure scales to hundreds of skills, but inaccurate descriptions can retrieve the wrong skill or miss a relevant one.
  • Pattern-2: Code-as-Skill: Code-as-skill represents capabilities as executable programs, offering deterministic outputs that support traditional testing and verification.Examples include Python functions, shell scripts, and domain-specific programs invoked through a runtime interface.
  • Pattern-3: Workflow Enforcement: Workflow-enforcement skills trade flexibility for reliability by constraining agents to prescribed action sequences with auditable steps.The pattern operates at the controller level and can host reusable skills from other patterns.

5.5. Pattern-4: Self-Evolving Skill Libraries

Self-evolving libraries generate, assess, and maintain skills after task execution, enabling library growth but making quality control central. Their effectiveness depends on domain specificity, deterministic verification, and generation-time quality gates.

  • Self-evolving libraries evaluate task trajectories and distill successful behavior into new or refined skills.Voyager validates generated code skills through in-game execution before adding them to a persistent library.
  • SkillsBench reports that self-generated skills average −1.3 pp relative to skill-free baselines, with improvement in only one of five tested configurations.
  • Self-generation succeeds in constrained environments with deterministic execution verification, whereas zero-shot generation without iterative verification can degrade open-ended performance.
  • Meta-skills can expand a small seed library without matching human effort by generating, modifying, or composing other skills at runtime.
  • Recursive skill generation can amplify errors, so quality gates are needed at each generation step.
  • Pattern-7: Marketplace Distribution: Marketplace distribution scales community skill creation but introduces supply-chain risk because packages may execute arbitrary actions within an agent’s permission scope.

5.9. Pattern Trade-offs

The patterns occupy a multidimensional trade-off space involving context cost, determinism, composability, and governance surface, and production systems commonly combine several patterns. The representation × scope taxonomy reveals both concentrated usage and substantial unexplored regions.

  • Context cost measures token consumption, determinism measures execution predictability, composability measures combination ease, and governance surface measures auditability and permission control.
  • Pattern Co-occurrence: Systems in the corpus use a median of 2 patterns, with a range of 1–4; Patterns 1+7 are the most common combination, appearing in 4 systems.
  • Pattern Co-occurrence: Production systems typically combine metadata-driven loading, marketplace distribution, hybrid implementation, and workflow enforcement for critical steps.
  • Computational Overhead: Retrieval, instruction loading, and multi-level composition impose overhead, while the latency–accuracy trade-off across deployment scenarios remains an open empirical question.
  • Representation × Scope Taxonomy: The representation × scope taxonomy separates how a skill policy is encoded from the environment or task domain in which it operates.
  • Representation × Scope Taxonomy: Most mapped systems occupy a sparse region centered on code-as-skill representations with SWE or web scope and self-evolving libraries, leaving other combinations underexplored.

6. Acquisition, Composition, Orchestration

The section surveys how agents acquire, refine, and compose reusable skills, from human authorship and demonstrations to self-practice and self-evolving libraries. Runtime orchestration uses retrieval or LLM routing, hierarchical decomposition, and recovery, while acquisition autonomy trades off against skill quality.

  • Acquisition: Agents acquire skills through human authorship, demonstration distillation, self-practice, curriculum-based learning, feedback, and meta-skill generation.These modes span decreasing human involvement and increasing autonomy.
  • Acquisition: Human-authored skills are easier to validate and audit but require labor to create, test, and maintain.The resulting skills are grounded in domain expertise and can be audited before deployment.
  • Acquisition: Self-practice enables continual learning without human supervision but may produce locally optimal, exploitative, or otherwise suboptimal procedures without external verification.This quality risk also applies to self-evolving libraries, where early errors can propagate through later generations.
  • Acquisition: Curriculum-based acquisition improves sample efficiency by training on progressively harder, ordered skill sequences rather than random ordering.Skill-it! provides the cited theoretical framework for this curriculum design.
  • Composition and Orchestration: Complex tasks are orchestrated by hierarchically decomposing goals, routing subtasks to specialized skills, and sequencing their execution through a planning layer.Runtime selection uses embedding retrieval, LLM-mediated routing, or a hybrid two-stage strategy balancing recall and precision.
  • Composition and Orchestration: When applicable skills conflict or terminate unsuccessfully, current systems lack explicit conflict-resolution policies and may invoke recovery skills to retry, backtrack, or switch strategies.Failure recovery can trigger re-retrieval or alternative skill selection.

7. Security, Trust, and Governance of Skills

The skill layer creates distinct security risks because skills combine instructions or code with agent capabilities and distribution channels. The section organizes defenses around trust tiers, sandboxing, provenance, auditing, and pattern-specific mitigation, illustrated by the ClawHavoc marketplace attack.

  • Threat Model: Compromised skills can steer agents toward malicious outcomes while appearing benign in metadata, creating a skill-specific attack surface.Threats include poisoned retrieval, malicious payloads, prompt injection, and confused-deputy behavior.
  • Trust Tiers: A four-tier trust model ranges from metadata-only discovery to autonomous execution, with progressively greater access to instructions and actions.T1 exposes only metadata; T2 exposes instructions; T3 supervises actions; T4 permits autonomous execution within configured boundaries.
  • Trust Tiers: Untrusted skills should default to Tier-1, while higher-tier access requires explicit escalation, provenance verification, monitoring, and demotion after safety violations.The proposed model makes trust sticky across reliable invocations but reversible after a violation.
  • Mitigations: Code skills require sandboxing, whereas natural-language skills require instruction–data separation, structured interfaces, and output filtering because context boundaries are permeable.Sandboxing granularity may be per-skill, per-session, or per-tier.
  • Governance: Marketplace governance should combine provenance signing, dependency auditing, automated scanning, and skill-native auditing to address supply-chain and natural-language attacks.Defense in depth combines binary scanning with tuple-level auditing of the skill’s components.
  • ClawHavoc Case Study: 36.8% of published skills contained at least one security flaw in the ClawHavoc case, whose attacks used cloned names, malicious code, prompt injection, and overbroad metadata.The campaign exploited marketplace distribution, code execution, documentation, and metadata-driven discovery.

8. Evaluating Agentic Skills

The section evaluates skills across correctness, robustness, generalization, safety, and related dimensions using deterministic, outcome-based verification. Benchmark coverage is incomplete, and SkillsBench provides direct but preliminary evidence that curated skills help while self-generated skills can hurt.

  • Evaluation Framework: Agentic skills are evaluated across correctness, robustness, generalization, and safety, with benchmarks mapped to the dimensions they assess.No single benchmark covers all dimensions, so comprehensive evaluation requires combining multiple benchmarks.
  • Evaluation Framework: Deterministic evaluation harnesses automatically compare environment state with expected outcomes, enabling reproducible, low-cost assessment integrated into skill development.Outcome-based verification prioritizes achieved effects over intermediate reasoning or implementation elegance.
  • SkillsBench: Focused skills with 2–3 modules yield +18.6 pp improvement, while 4+ skills yield +5.9 pp and comprehensive skills degrade performance by −2.9 pp.Detailed skills improve by +18.8 pp, indicating that quantity and documentation complexity matter.
  • Interpretation: The benchmark findings are interpretive hypotheses rather than causal conclusions and require independent replication across additional benchmarks.Other cited results provide partial corroboration but do not directly measure the curated-versus-self-generated comparison.

9. Discussion and Limitations

The discussion identifies a tension between skill representations, governance, and scalability. It also emphasizes that the surveyed ecosystem is recent and incomplete, so the proposed patterns and conclusions may change as evidence accumulates.

  • Discussion: More formal representations support stronger governance, but natural-language skills are easier to author while code and policy require specialized expertise.No existing system fully resolves this representation–governance tension; hybrid representations remain part of the design space.
  • Discussion: Curated skills improve pass rates by +16.2 pp on average, while self-generated skills degrade them by −1.3 pp, exposing a curation–scalability tradeoff.Human curation is more reliable but does not scale, whereas self-evolving libraries scale but can hurt performance.
  • Limitations: The LLM-specific literature may be too nascent for the identified patterns to be stable, and the taxonomy may require revision as the field matures.The analysis examines 24 systems from a retained set of 65 papers.

10. Open Problems and Research Roadmap

Reliable skill-based agents face unresolved tensions in verification, discovery, representation, environmental maintenance, and marketplace governance. The roadmap emphasizes admission-time evaluation, more autonomous discovery, layered auditing, drift monitoring, and accountability mechanisms.

  • Verified Autonomous Skill Generation: Autonomous skill generation can expand capability libraries rapidly, but self-generated skills may degrade downstream performance while human curation limits scalability.The paper frames this as a central scalability–reliability trade-off.
  • Verified Autonomous Skill Generation: Held-out evaluation before library admission is proposed as a verification pipeline for newly generated reusable skills.Code-centric skills may additionally benefit from formal or semi-formal verification.
  • Autonomous Skill Discovery: Fully autonomous skill discovery remains rare because most systems depend on curricula, demonstrations, reward signals, or other human guidance.Unsupervised discovery from interaction traces is suggested as one possible path beyond supervised discovery.
  • Representation and Verification: Natural-language and policy-style skills are easier to author and distribute than executable code, but are significantly harder to audit rigorously.The paper identifies a mismatch between expressive convenience and verifiability.
  • Environmental Drift: Environmental drift can invalidate skill assumptions when APIs, tools, data formats, or workflows change, even without modifying the skill artifact.Continuous monitoring could detect behavioral deviations and support adaptation or retirement.
  • Governance Economics and Liability: Open skill marketplaces broaden contribution and innovation while expanding supply-chain risks, with unclear responsibility and weak certification mechanisms.The roadmap calls for liability models and certification that align incentives with reliability.

11. Conclusion

The paper presents agentic skills as reusable procedural modules for LLM agents, structures their design space, analyzes their security risks, and links skill quality to agent performance. It closes by identifying discovery, verification, and governance as open challenges for reliable skill-based agents.

  • 11. Conclusion: Agentic skills are reusable procedural modules for LLM agents, and the paper organizes their design space and security risks.The conclusion also identifies skill quality as critical to agent performance.
  • 11. Conclusion: Discovery, verification, and governance remain open challenges for building reliable skill-based agents.
Loading 2602.20867v1…