PaperlayerView exact revision on arXiv

Source-linked AI summary

Demystifying Agent Skills for Smart Contract Auditing: Design, Effectiveness, Behavioral Impact

Cuifeng Gao, Juantao Zhong, Jiachi Chen, Shuai Wang, Daoyuan Wu

arXiv:2609.29454v1cs.SE

TL;DR

Smart-contract audit skills remain underexplored despite their relevance to agent auditing. We curate 83 skills and evaluate them on EVMBench across seven agent–model configurations, finding that gains depend mainly on backend capability and that skill triggering limits behavioral impact.

  • Problem

    Evidence remains limited on how smart-contract audit skills are designed and affect agent effectiveness and behavior, while prior skill studies often provide little domain-specific insight.

    Source: §I · p. 1

  • Method

    We curate 83 smart-contract audit skills and evaluate them on EVMBench across seven agent–model configurations, studying design, detection effectiveness, and execution trajectories.

    Source: §I · p. 1

  • Results

    Codex/GPT-5.5 gains +22.8% Detect Score and +43.2% Detect Award; skill effectiveness depends primarily on backend capability, and triggering remains the behavioral bottleneck.

    Source: §I · p. 2

  • Takeaways & Limitations

    When invoked, audit skills preserve a shared core audit workflow while producing distinct loading patterns and configuration-dependent behavioral effects.

    Source: §I · p. 2, §IX · p. 11

    The findings may not generalize to all benchmarks, models, or skill ecosystems.

    Source: B. Threats to Validity · p. 10

Abstract

from arXiv · show

LLM agents, notably Claude Code and OpenAI Codex, are emerging as versatile tools beyond coding agents only. These agents can be enhanced with skills---reusable artifacts that package domain knowledge, workflows, and tool-use instructions. To date, however, little is known about how such skills are designed or how they affect agent effectiveness and behavior in practice. In this paper, we investigate these questions in smart contract security auditing, a domain in which agents have shown substantial promise. We systematically collect 83 smart contract audit skills from the wild and evaluate them on EVMBench across seven agent--model configurations. Our study examines three dimensions: (i) the design characteristics of audit skills, including their structure, knowledge representations, workflows, and tool dependencies; (ii) their effectiveness in improving vulnerability detection; and (iii) their influence on agent execution trajectories. We find that audit skills are mostly lightweight but heterogeneous in design, covering a broad yet imbalanced range of vulnerability types. Their effectiveness is determined primarily by the model rather than the agent harness: Codex/GPT-5.5 achieves the largest gains, improving detection score by 22.8% and captured award by 43.2%. We further find that skill triggering is a key bottleneck. When triggered, skills preserve a shared six-stage audit workflow while exhibiting distinct loading patterns and differential effects on agent behavior across configurations. We release our skill corpus and artifacts to support future research.

I. INTRODUCTION

The paper addresses limited evidence about how smart-contract audit skills are designed and affect agent effectiveness and behavior. It studies 83 collected skills and finds that benefits depend strongly on backend capability and whether agents trigger skills.

I. INTRODUCTION

  • Agent skills package domain knowledge, workflows, tool-use instructions, vulnerability taxonomies, and validation criteria as reusable artifacts for guiding agent behavior.Smart-contract auditing requires repository understanding, domain-specific reasoning, tool-assisted analysis, and careful validation of suspected vulnerabilities.Source: §I · p. 1
  • Prior work offers limited insight into audit-skill design and effects, while smart-contract security research has mainly examined agents’ native auditing, repair, and exploitation capabilities.Existing skill studies often remain black-box and provide limited domain-specific insight.Source: §I · p. 1
  • The study collects and curates 83 smart-contract audit skills and evaluates them on EVMBench across seven agent–model configurations.It examines skill design, vulnerability-detection effectiveness, and agent execution trajectories.Source: §I · pp. 1–2
  • Audit skills are mostly lightweight and instruction-centric, but heterogeneous in knowledge representation, tool dependencies, workflow design, and vulnerability coverage.They combine heuristic, procedural, reporting, conceptual, and example-based knowledge; classical risks are widely covered, while protocol-specific and DeFi risks form a fragmented long tail.Source: §I · p. 2
  • +22.8% Detect Score and +43.2% Detect Award are the largest reported gains, achieved by Codex/GPT-5.5; skill benefits depend primarily on backend capability.Only Codex/GPT-5.5 activates skills consistently across all audits; skills shift trajectories in six of seven configurations.Source: §I · p. 2

II. PRELIMINARIES

Agent skills are folders with a required instruction file and optional resources, loaded through discovery, activation, and execution. The study gathers audit skills from multiple source types, curates them, and finds the final corpus remains semantically diverse.

II. PRELIMINARIES

  • A skill is a folder containing a mandatory SKILL.md file and optional resources; its instructions load in stages as the agent discovers, activates, and executes a skill.Discovery loads metadata, activation loads the complete instructions when a task matches, and execution may access bundled resources.Source: §II · p. 2
  • The study focuses on smart-contract audit skills because the domain combines consequential auditing, rich skill artifacts, and real-world skills with realistic benchmarks.This setting supports analysis beyond aggregate performance, including skill design, activation, and execution-level mechanisms.Source: §II · p. 2

III. AUDIT SKILLS COLLECTION

  • Candidate skills come from AI skill marketplaces, community-maintained GitHub collections, and academic benchmarks.Marketplace searches use the phrase “smart contract security audit”; the academic dataset is filtered by blockchain- and security-related terms.Source: §III · p. 3
  • Skills are retained only if they target Solidity or EVM contracts, support security auditing or vulnerability detection, and meet the required SKILL.md format.Necessary resources must be self-contained in the skill directory.Source: §III · p. 3
  • After exact and manual near-duplicate removal, the curated corpus contains 83 unique smart-contract audit skills.The process reduces 155 candidates to 131 unique URLs, then 120 skills after exact deduplication, before manual curation.Source: §III · p. 3
  • 86.5% of the 3,403 skill pairs have cosine similarity below 0.75, while 30 pairs (0.9%) reach at least 0.85.The authors interpret this distribution as evidence that the final corpus retains diverse audit instructions and knowledge representations.Source: §III · p. 3

IV. RQ1: SKILL DESIGN

The design analysis measures structural properties and characterizes skill content across knowledge representation, tool dependencies, workflow design, and vulnerability coverage. It uses corpus statistics and bottom-up analysis of skill contents.

IV. RQ1: SKILL DESIGN

  • The design analysis measures structural properties such as skill scale and composition, then examines four content dimensions: knowledge representation, tool dependencies, workflow design, and vulnerability coverage.The dimensions are guided by questions in Anthropic’s skill design guideline.Source: §IV · pp. 3–4
  • The analysis pipeline derives structural characteristics from the 83-skill corpus and examines skill contents through bottom-up open coding.An agent incrementally reads each repository to extract grounded observations, followed by AI-assisted taxonomy induction, category assignment, and structural profiling.Source: §IV · p. 4, Figure 3 · p. 4
  • GPT-5.4 performs the primary analysis, Claude Opus 4.6 independently reviews intermediate outputs, and researchers manually adjudicate remaining ambiguities.The review loop is capped at 15 rounds.Source: §IV · p. 4

A. RQ1.1 Structural Characteristics

Audit skills are generally lightweight, instruction-centric artifacts, with supporting resources commonly organized around reusable references. The study also derives a taxonomy for how audit knowledge is represented.

A. RQ1.1 Structural Characteristics

  • The median SKILL.md contains 120 lines and 1,281 tokens, while the median artifact is 25.43 KB with 3 files; reported averages are 189 lines, 16 files, and 121 KB.The median and average figures are distinct corpus summaries.Source: A. RQ1.1 Structural Characteristics · p. 4
  • References is the most common bundled directory, appearing in 44 skills, followed by templates in 16 and scripts in 9.Fig. 4 shows the distribution of bundled resource directory categories.Source: A. RQ1.1 Structural Characteristics · p. 4, Figure 4 · p. 4

B. RQ1.2 Knowledge Representation

C. RQ1.3 Tool Dependency

Most audit skills include operational tool support, but vulnerability-analysis dependencies occur in a smaller subset. Their workflows tend to be staged and checkpoint-driven, with explicit validation and report-oriented delivery.

C. RQ1.3 Tool Dependency

  • Across the corpus, 73 of 83 skills (88.0%) are tool-augmented; among smart contract audit skills, 23% carry vulnerability-analysis tool dependencies, primarily static analysis, often complemented by symbolic execution.Source: C. RQ1.3 Tool Dependency · p. 5
  • Static analysis appears in all 19 skills with vulnerability-analysis dependencies, compared with symbolic execution in 12, fuzzing in 7, and formal verification in 6.Slither appears in 18 skills and Mythril in 10; auxiliary tools support parsing, semantic reasoning, reverse engineering, or evaluation.Source: C. RQ1.3 Tool Dependency · p. 5

D. RQ1.4 Workflow Design

  • The median workflow has 5 stages, 3 decision points, and 6 execution steps, while 73 skills (88.0%) have 4–6 stages.Fig. 5 summarizes the workflow profile.Source: Figure 5 · p. 5, D. RQ1.4 Workflow Design · p. 5
  • Workflows commonly use single-threaded coordination (62 skills), explicit validation (46), report-oriented delivery (51), and milestone gates (58), while parallel delegation and iteration are rare.Activation is often feature- or user-triggered; 28 skills use protocol-feature triggers and 17 use explicit user inputs or command arguments.Source: D. RQ1.4 Workflow Design · pp. 5–6, Table V · p. 5

E. RQ1.5 Vulnerability Coverage

The corpus covers many vulnerability categories, but coverage is uneven: classical contract risks dominate while protocol-specific and DeFi-oriented risks form a long tail. Individual skills also differ substantially in coverage breadth.

E. RQ1.5 Vulnerability Coverage

  • The study maps 2,482 vulnerability items from 83 skills to a manually validated taxonomy of 37 atomic vulnerability categories.Coverage is analyzed both by how many skills cover each category and by how many categories each skill covers.Source: E. RQ1.5 Vulnerability Coverage · p. 6
  • Reentrancy (41 skills), arithmetic precision (40), and access control (33) are the most widely covered categories, while protocol-specific and DeFi risks form a less-covered long tail.Examples in the long tail include MEV, flashloan exploits, and governance manipulation; Fig. 6 presents the category coverage distribution.Source: E. RQ1.5 Vulnerability Coverage · p. 6, Figure 6 · p. 6
  • Individual skills cover 1–26 vulnerability categories, with a median of 9, ranging from broad-spectrum auditors to narrowly specialized skills.Fig. 7 shows the distribution of coverage breadth across skills.Source: E. RQ1.5 Vulnerability Coverage · p. 6, Figure 7 · p. 6

V. RQ2: SKILL EFFECTIVENESS

Audit skills improve Detect Score across all four flagship configurations, with Codex/GPT-5.5 showing the largest gain and improving both cost and runtime. The evaluation compares skill availability on EVMBench across agent–model configurations, leaving skill discovery and selection to the agents.

V. RQ2: SKILL EFFECTIVENESS

  • The evaluation uses EVMBench’s 40 real-world audit tasks and 120 annotated vulnerabilities, pairing four harnesses with their native flagship models and adding a same-backend DeepSeek-V4-Pro setting.Antigravity is excluded from the same-backend setting because its CLI lacks an API-configurable model backend.Source: §V · p. 6, A. Experimental Setup · p. 6

A. Experimental Setup

  • Skills are globally available in the with-skills condition, but agents are not instructed to invoke them; task instructions otherwise remain identical to the no-skills baseline.The comparison is designed to isolate the contribution of skill availability.Source: A. Experimental Setup · p. 7
  • Effectiveness is assessed through Detect Score and Disclosure Precision, alongside Detect Award and runtime and token consumption.Detect Score counts correctly detected vulnerabilities; Detect Award measures their ground-truth monetary award value.Source: A. Experimental Setup · p. 7

B. Detection Results and Analysis

  • 3.2%–22.8% relative Detect Score gains occur across flagship configurations; Codex/GPT-5.5 gains 22.8% in Detect Score and 43.2% in Detect Award, with non-uniform overhead.Codex/GPT-5.5 also reduces token cost per run by 5.24% and runtime by 3.64%; Fig. 8 presents detection cost under the flagship setting.Source: B. Detection Results and Analysis · pp. 7–8, Figure 8 · p. 8

C. Ablation Study under the Same Weaker Backend Model

The same-backend ablation indicates that skill effectiveness depends more on backend model capability than harness design. Under DeepSeek-V4-Pro, skills provide little or negative benefit, with only mild computational overhead.

C. Ablation Study under the Same Weaker Backend Model

  • Across three harnesses, audit-skill effects become negative under DeepSeek-V4-Pro, ranging from -5.9% to -9.4%.The study identifies backend model capability as a key determinant of whether agents can effectively leverage audit skills.Source: C. Ablation Study under the Same… · p. 8
  • With DeepSeek-V4-Pro held constant, harnesses show similar skill effectiveness, with variations generally within 11%.The with-skills condition trends downward across the harnesses.Source: C. Ablation Study under the Same… · p. 8
  • Token cost varies from −6.35% to +0.91%, while runtime decreases across harnesses by −3.07% to −25.14%.These are the reported computational overhead changes under the DeepSeek-V4-Pro setting.Source: C. Ablation Study under the Same… · p. 8

VI. RQ3: BEHAVIORAL IMPACT

The behavioral analysis tests whether skills shift audit trajectories, whether agents invoke them, and how invocation affects workflows. Six of seven configurations show significant shifts in at least one view, while skill activation is concentrated in Codex/GPT-5.5.

VI. RQ3: BEHAVIORAL IMPACT

  • The study compares with-skill and without-skill trajectories using paired semantic and behavioral views, then tests paired shifts with Hotelling’s T 2 and sign-flip permutation tests.The permutation test uses B=5000; semantic trajectories use text embeddings, while behavioral trajectories are 10-dimensional action-type share vectors.Source: §VI · p. 8, A. Skill-Induced Trajectory Shifts · p. 8

A. Skill-Induced Trajectory Shifts

  • Six of seven configurations show significant trajectory shifts in at least one view; Codex/GPT has p = 0.0002 in both views, while Claude/DeepSeek is non-significant in both.The two views capture semantic reasoning content and behavioral action shares.Source: A. Skill-Induced Trajectory Shifts · pp. 8–9

B. Skill Triggering

  • Only Codex/GPT-5.5 activates skills consistently, while other configurations trigger them sporadically or not at all; activation is primarily associated with model capability.At most 27 of 83 available skills are invoked, and total skill reads range from 0 to 119.Source: B. Skill Triggering · p. 9

C. Trajectory Impact of Skill Invocation

Skill invocation occurs within a shared six-stage audit workflow, but its timing and usage pattern vary across configurations. Downstream trajectory footprints also differ, from broad changes across design dimensions to limited or inconsistent changes.

C. Trajectory Impact of Skill Invocation

  • Across configurations, audits retain a six-stage backbone from instruction reading through report generation, while skill use occurs early or after substantial code exploration.Codex/GPT-5.5, OpenCode/DeepSeek-V4-Pro, and Antigravity/Gemini-3.5-Flash tend to load skills early; OpenCode/GLM-5.1 tends to do so later.Source: C. Trajectory Impact of Skill Invocation · p. 9, Figure 11 · p. 9
  • Agents use either one primary workflow skill or chain multiple skills across audit stages, with chained use described as context-aligned and stage-wise.The paper reports single-skill use for Antigravity/Gemini-3.5-Flash and OpenCode/DeepSeek-V4-Pro, and chained use for Codex/GPT-5.5 and OpenCode/GLM-5.1.Source: C. Trajectory Impact of Skill Invocation · p. 9
  • Codex/GPT-5.5 shows footprints across knowledge, tool, and workflow dimensions, while Antigravity/Gemini-3.5-Flash changes mainly tool use and the OpenCode configurations show no consistent footprint.The mapped dimensions include structured checklists and incident-based reasoning, analyzer use, and skill-prescribed workflow scaffolds.Source: Table VIII · p. 10, C. Trajectory Impact of Skill Invocation · p. 10

A. Robustness Analysis

Repeated Codex/GPT-5.5 runs preserve the direction of skill-related improvements. The paper also notes limits to causal attribution and generalization, situates its study against domain-agnostic skill research, and concludes that model capability and skill triggering shape observed impact.

A. Robustness Analysis

  • Across three repeated Codex/GPT-5.5 runs, skills consistently improve Detect Score and Award, while Precision remains relatively stable.The direction of improvement is preserved in every run, though most other results use single-run executions.Source: Table IX · p. 10, A. Robustness Analysis · p. 10

B. Threats to Validity

  • Most results are single-run executions, and autonomous invocation does not isolate each skill’s effect; findings may not generalize across benchmarks, models, or skill ecosystems.The corpus is a temporal snapshot, and the study evaluates a representative rather than exhaustive set of models; repeated Codex/GPT-5.5 runs mitigate stochasticity concerns for that configuration.Source: B. Threats to Validity · p. 10

VIII. RELATED WORK

  • Prior skill research is largely domain-agnostic and often black-box, while smart-contract audit skills remain underexamined in design, effectiveness, and behavioral impact.The paper presents its study as the first systematic empirical study of smart contract audit skills.Source: §VIII · pp. 10–11

IX. CONCLUSION

  • The study evaluates 83 curated audit skills across seven agent–model configurations on EVMBench and concludes that skill benefits depend mainly on backend model capability, with triggering as a key bottleneck.When invoked, skills preserve the core audit workflow.Source: §IX · p. 11
Loading 2609.29454v1…