PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Maliciousness: Static and Runtime Attack-Surface Profiling of Agent Skills

Ratnesh Pandey, Japneet Singh, Harsh Akshit, Sonali Tyagi, Cassie Crossley

arXiv:2610.19264v1cs.CRcs.AI

TL;DR

Agent-skill research has largely measured malicious intent, leaving capability exposure across the same marketplace population less directly characterized. This paper profiles bundles with static and runtime evidence, then finds that consequential capabilities are common even where threat verdicts are narrower. Its findings position capability profiling as a complement to maliciousness detection, while the sampled corpus limits marketplace-wide generalization.

  • Problem

    Prior agent-skill studies had not separately operationalized malicious intent and capability exposure over the same marketplace population or reported their joint distribution.

    Source: §VI · p. 11

  • Method

    The study combines static bundle analysis and provenance checking with instrumented sandbox execution, assessing static and runtime evidence as complementary views.

    Source: 2) What is measured · p. 4, 5) Provenance analysis · p. 5, 2) The phases see different kinds of… · p. 10

  • Results

    Seven in ten listings carry at least one elevated capability, while the threat verdict marks a smaller population; at the top two exposure levels, benign listings outnumber flagged ones four to one.

    Source: §I · p. 2, §VI · p. 11

  • Takeaways & Limitations

    Capability-oriented attack-surface profiling complements malicious-intent verdicts by identifying exposure that those verdicts do not capture.

    Source: §VI · p. 11

Abstract

from arXiv · show

Agent skills are Markdown documents that a tool-using agent reads as instructions and then acts on with the privileges of the session that loaded them. Installing one grants authority (the standing to direct the agent, not new privilege), and the size of that grant is what we measure. We present a two-phase attack-surface profiling methodology that combines whole-bundle static analysis and dependency provenance checking with instrumented sandbox execution and separate model-based assessment of the resulting traces. We apply the methodology to 6,309 listings from two marketplaces, selecting listings for execution without conditioning that decision on a prior threat or risk signal. Half the corpus carries evidence of shell-execution capability; the corresponding shares are 35% for filesystem read, 31% each for filesystem write and network egress, and 13% for credential access; seven listings in ten carry at least one elevated capability. That is the surface a user accepts at install time, and almost none of it is visible then. A permission model exists but is close to inert. Authors may list the tools a skill pre-approves in its own frontmatter, which removes confirmation prompts rather than restricting what the skill can reach; of the 6,061 listings whose frontmatter parses, 770 declare anything at all, the vocabulary cannot express credential or environment access, and nothing checks what it does express against the bundle. Declaring is also not declaring accurately: 59% of declaring listings understate what their bundle can do. ClawHub declares roughly twice as often as skillsmp, yet under-declares on 88% of its declarations against 31%. Triage at the marketplace boundary therefore cannot rest on what a listing declares. It must rest on what the bundle can do. These results position capability-oriented attack-surface profiling as a necessary complement to maliciousness detection.

I. INTRODUCTION

Agent skills act within a session’s existing authority, yet their install-time declarations neither restrict that authority nor reliably describe it. This paper distinguishes capability exposure from malicious intent, highlighting how much of a skill’s consequential authority may remain invisible before installation.

I. INTRODUCTION

  • Installation places a skill’s instructions and scripts in the agent’s existing session, where it can reach the shell, filesystem, network, and credentials already held by that session.The optional frontmatter tool list can remove confirmation prompts, but it is a grant rather than a restriction and travels with installations.Source: §I · p. 1
  • A frontmatter declaration is optional, author-written, unchecked against bundle contents, and unable to express credential or environment access in a machine-checkable field.Even an accurate tool declaration cannot establish the limits of actions performed through a shell or invoked script.Source: §I · p. 1

A. Threat Measurement and Its Limits

  • Earlier marketplace research documented rapid skill growth and malicious skills, but primarily asked whether a listing was trying to cause harm rather than what authority it exposed.The cited studies include static analysis of 31,132 skills, a detonation study, and reports of live campaigns.Source: §I · p. 2

I. INTRODUCTION

  • 70% of listings carry at least one elevated capability, including shell execution in half, filesystem reads in 35%, filesystem writes and network access in 31% each, and credential access in 13%.Fig. 1 contrasts the bundle-inferred capability footprint with the smaller populations judged to carry serious exposure or a threat verdict.Source: Figure 1 · p. 1, §I · p. 2
  • The paper distinguishes bundle-inferred capability and exposure from malicious intent; none of these measures alone implies observed exploitation or realized harm.The study defines authority as standing to direct the agent without per-action confirmation, not as new privilege.Source: §I · p. 2

A. Threat Measurement and Its Limits

  • Prior studies have measured malicious intent, but the cited work had not separately operationalized intent and capability exposure over the same marketplace population or reported their joint distribution.The cited comparison studies examined scanner agreement or exposure under adversarial configuration search, rather than default-installation exposure.Source: A. Threat Measurement and Its Limits · p. 2

B. From Static Analysis to Detonation

The paper combines whole-bundle and runtime evidence to profile skill attack surfaces, while examining declaration gaps and mapping findings to security frameworks. It distinguishes its unfiltered execution cohort from prior candidate-selected detonations and treats static and runtime evidence as complementary.

B. From Static Analysis to Detonation

  • Static and runtime phases observe different evidence: bundle text can reveal unpinned imports, while detonation can expose destinations reached only under particular inputs.Earlier work also detonated skills and other artifacts; the paper makes no claim to be first or largest.Source: B. From Static Analysis to Detonation · p. 2
  • All listings in the sampled cohort were selected for detonation without prior threat- or risk-based filtering, unlike cited agent-era studies that selected candidates after earlier analysis.The cohort is popularity-banded, not a complete census or probability-representative sample, so its exposure estimates are not marketplace-wide base rates.Source: B. From Static Analysis to Detonation · p. 2

C. Capability Declaration

  • Unlike ecosystems where permissions are enforced and over-requesting dominates, agent-skill declarations more often understate bundle capabilities because the field restricts nothing.Prior work measured permission over-request, while this paper situates skill declarations among proposals for additional capability and environment fields and related measurements.Source: C. Capability Declaration · p. 3

D. Framework Mapping

  • The framework mapping covers MITRE ATLAS, the OWASP Agentic Top 10, and the OWASP Agentic Skills Top 10, with coverage figures reported for the least mature catalog.Several mitigations in that catalog presuppose an inspectable permission manifest, which the paper measures in its declaration analysis.Source: D. Framework Mapping · p. 3

E. Contributions •

  • The methodology combines whole-bundle static analysis and dependency provenance checking with instrumented sandbox execution and separate model-based assessment of traces.Source: E. Contributions • · p. 3
  • The study applies the method to 6,309 unfiltered marketplace listings to analyze threat and exposure independently, including substantial capability exposure without measured malicious intent.It also measures declaration adoption, expressiveness, and agreement between author pre-approvals and inferred capabilities.Source: E. Contributions • · p. 3
  • The empirical comparison finds that static and runtime evidence identify materially different aspects of skill risk, while threat and exposure findings concentrate in different framework categories.Source: E. Contributions • · p. 3

III. SCOPE AND METHODOLOGY

The study samples listings from two marketplaces and profiles them through whole-bundle static analysis, sandbox execution, and human review. Its scope and review coverage constrain how the resulting measurements should be interpreted.

A. Scope

  • Publisher concentration is uneven: one publisher contributes 266 listings, and skillsmp has more listings per publisher on average than ClawHub.The averages are 4.1 listings per publisher on skillsmp versus 1.7 on ClawHub.Source: A. Scope · p. 3
  • 6,309 listings from two marketplaces form a point-in-time sample, not a complete census; popularity is measured with different proxies across marketplaces.The study sampled by star or download-count bands and excluded the popularity extremes.Source: A. Scope · p. 3

B. Methodology

  • The methodology combines whole-bundle static inspection with instrumented sandbox execution, with sampled human review qualifying automated findings.Targeted re-scans also supplement automated findings.Source: B. Methodology · p. 4, 1) Two tiers · p. 4

1) Two tiers:

  • The human review cohort covers 410 listings from one marketplace, and its in-progress reviews support no reported precision figure.It includes 91 flagged and 319 unflagged listings; unflagged listings are sampled.Source: 1) Two tiers · p. 4
  • 140 randomly selected benign listings were read in full, with none identified as a missed threat; the estimated benign miss-rate upper bound is approximately 2.1%.This bound is not a precision figure.Source: 1) Two tiers · p. 4
  • Every claimed missed threat and borderline case is re-verified against bundle bytes, while severity increases generally apply automatically and decreases require human review.Increases based only on a known false-positive pattern are also withheld.Source: 1) Two tiers · p. 4

2) What is measured:

The paper measures malicious intent, exposure severity, and capability footprint as distinct quantities. It derives findings from whole-bundle static analysis and sandbox detonation, while separating the phases’ evidence and correcting runtime aggregation for common executor activity.

2) What is measured:

  • Attack surface describes the bundle’s capability footprint before severity or intent is assigned; threat and exposure are scored on separate axes.Threat verdicts use thresholds of 0.70 for malicious and 0.50 for suspicious, while risk bands an exposure score from zero to ten.Source: 2) What is measured · p. 4

3) Static analysis:

  • The static phase reads complete bundles and supplements model findings with four deterministic heuristics; it produces 4,246 findings.The heuristics cover hard-coded secrets, piped installers, invisible Unicode, and opaque binaries.Source: 3) Static analysis · pp. 4–5
  • A keyword crosswalk groups analyst finding titles into named patterns but adds no detections; it matches 95% of static threat findings and 87% of static risk findings.A stratified sample of 74 assignments was audited by hand.Source: 3) Static analysis · p. 5
  • The 4,246 static findings partition between two single-code schemes, while MITRE ATLAS technique mentions are not comparable as a finding count.The schemes route 2,508 findings to Agentic Skills Top 10, 1,710 to Agentic Top 10, and 28 to neither; ATLAS records 3,789 mentions across fifteen techniques.Source: 3) Static analysis · p. 5

4) Detonation:

  • Nearly all listings were detonated in an isolated sandbox, and a separate judge model assessed the resulting process, tool-call, and network-egress traces.The static analyst, detonating agent, and judge use three configurations; the phases differ by vendor as well as prompt and evidence.Source: 4) Detonation · p. 5
  • Runtime aggregation drops names appearing in at least half of listings and four named executor runtimes, leaving 522 process names distinctive to some subset.This is a cohort-level prevalence correction, not subtraction of each listing’s own control run.Source: 4) Detonation · p. 5

5) Provenance analysis:

Dependency provenance checks whether referenced dependencies, repositories, and publishing accounts resolve, but do not establish intent. The findings show elevated capabilities are common, while model-based measurements have scope and reproducibility limits.

5) Provenance analysis:

  • Provenance analysis resolves listed dependencies, repositories, and publishing accounts and checks dependency pinning and takeover-prone hosts, but its denominator is listings declaring dependencies.Resolution establishes existence, not whether the resolved item is the one the author intended; lookalikes can resolve, and name-similarity detection is not performed.Source: 5) Provenance analysis · p. 5

6) Limits of the measurement:

  • Verdict reproducibility was 84% across 89 re-scans, aggregate rates have no uncertainty intervals, and publisher concentration makes listings non-independent.One publisher contributes 266 listings; disagreement does not establish that errors balance or cancel.Source: 6) Limits of the measurement · p. 5
  • Sandbox observations bound runtime capability evidence from below, and most listings ran once without a control; capability and amplification labels are canonicalized model-emitted text.Unreferenced setup scripts that write persistent configuration may never execute, and control runs showed shell invocation and environment access in every listing in the smaller controlled cohort.Source: 6) Limits of the measurement · p. 5

A. High-Level Findings

  • 4,431 of 6,309 listings carry at least one elevated capability, compared with 3,089 carrying risk and 338 carrying a non-benign verdict.Capability is the common condition, exposure is frequent, and malicious intent is rare.Source: A. High-Level Findings · p. 5
  • Every listing with a threat verdict has medium-or-higher exposure, but four in five listings at high or critical exposure carry no threat verdict.There are 1,306 benign listings at those exposure levels versus 315 flagged listings; a hand-read random sample found no missed threat and bounded the benign miss rate at approximately 2.1%.Source: A. High-Level Findings · pp. 5–6
  • Only 770 of 6,061 parseable listings declare tools, and 457 of those declarations understate bundle capability; the two analysis phases also disagree on most flagged findings.The paper reports both phases because their findings largely differ.Source: A. High-Level Findings · p. 6

B. Threat Profiling

Threat verdicts are uncommon, but the threat patterns that do appear are concentrated in dual-use tooling rather than covert malware. Severity is concentrated at the top of the finding scale.

1) How much of the corpus carries a threat:

  • 338 listings (5.4%) carry a non-benign verdict: 206 malicious and 132 suspicious.Broadening the threshold to any threat-axis finding or lower-threshold threat score gives 345 listings; the pattern counts overlap.Source: 1) How much of the corpus carries a… · p. 6

2) What the threats are:

  • Roughly one-third of the 345 listings with any threat signal fall in the leading pattern, which exceeds the next four patterns combined.The leading pattern is offensive tooling packaged as a skill, dual-use by construction; excluding it leaves a long tail with no pattern reaching 40 listings.Source: 2) What the threats are · p. 6, Figure 3 · p. 6
  • 353 of 383 threat findings are critical or high, leaving 30 across the three lower severity levels.Severity is measured per finding, not per listing.Source: 2) What the threats are · p. 6

3) Standards mappings:

Threat and risk are distinct measurements: they map differently to standards and select populations of very different sizes. Risk is common, and it often appears without a threat verdict.

3) Standards mappings:

  • The skills-specific catalog maps 240 of 280 threat-axis findings to malicious skills, while broader schemes distribute mapped findings across several categories.The Agentic Top 10’s leading threat categories are identity and credential abuse (38), goal and instruction hijack (27), and tool misuse (23); MITRE ATLAS’s are external harms (215), unsecured credentials (55), prompt injection (38), and supply-chain compromise (37).Source: 3) Standards mappings · p. 6

4) Threat by marketplace and popularity:

  • Skillsmp’s broader threat-signal rate is 6.6% (260/3,947), versus 3.6% (85/2,362) on ClawHub, but different sampling prevents ranking their security programs.Threat rates across popularity bands show no trend on skillsmp (5.8%, 7.4%, 6.6%); ClawHub ranges from 4.6% to 2.0%, with only the endpoints separated.Source: 4) Threat by marketplace and popularity · pp. 6–7

1) How much of the corpus carries risk:

  • 3,089 listings (49.0%) meet the risk test; the exposure score is bimodal, with 3,128 listings scoring zero and 1,621 scoring at least seven.Among the 3,181 listings with a nonzero score, the mean is 6.81 and median 7.1; at least 1,250 listings have elevated capability but zero exposure, so capability presence and exposure magnitude differ.Source: 1) How much of the corpus carries risk · p. 7

2) Threat and risk on the same listings:

  • No listing with a threat verdict is in the low-exposure band, yet 1,306 benign listings are high or critical exposure, versus 315 flagged listings.Four in five listings at the top two exposure levels have no threat verdict; the measures select populations of different sizes.Source: 2) Threat and risk on the same listings · p. 7

3) What the risks are:

Risk patterns are more broadly distributed than threat patterns, while severity findings and listing-level exposure use different scales. Standards mappings also emphasize install-time bundle properties rather than attacker behavior.

3) What the risks are:

  • Sensitive data crossing a trust boundary leads the risk patterns at 809 listings, followed by agent authority expanding past the task (776) and unvetted third-party components (703).Six patterns exceed 180 listings, but counts are overlapping static-phase findings, not shares of the 3,089 listings meeting the risk test; the lethal trifecta appears in 628 listings and indicates a potential exfiltration path, not observed leakage.Source: 3) What the risks are · p. 7

4) Severity, and why two ladders disagree:

  • Only 2 risk findings are critical, compared with 1,262 high, 915 medium, 1,487 low, and 197 informational findings.Finding severity rates an observation; exposure levels aggregate capability reach and amplification, so a listing can reach critical exposure without an individually critical finding.Source: 4) Severity, and why two ladders… · pp. 7–8

5) Standards mappings:

  • Over-privileged skills and supply-chain compromise account for over four fifths of risk findings mapped under the Agentic Skills Top 10, unlike their limited threat-axis presence.The mapping counts are 1,051 and 799, respectively; the Agentic Top 10 instead leads with goal and instruction hijack (752), tool misuse (376), and identity and credential abuse (212).Source: 5) Standards mappings · p. 8

6) Risk by marketplace and popularity:

The marketplace rankings reverse across risk and threat, and the threat comparison is sensitive to a recent change in ClawHub coverage. Capability and amplification profiles describe what listings may enable, not intent.

6) Risk by marketplace and popularity:

  • ClawHub has a higher risk share than skillsmp (65.6% versus 39.0%), while the broader threat-signal ordering reverses (3.6% versus 6.6%).The risk ordering held across corpus revisions, but ClawHub’s threat rate fell from 6.3% among 862 listings to 3.6% among 2,362; marketplace controls were not evaluated, and the comparison does not establish their effectiveness.Source: 6) Risk by marketplace and popularity · p. 8

1) Inferred capability footprint:

  • 4,431 listings (seven in ten) carry at least one elevated capability, and half the corpus has inferred shell-execution capability.Listings average 1.9 capabilities and 1.6 amplification factors; 397 have both credential access and network egress, while 628 meet the lethal-trifecta description.Source: 1) Inferred capability footprint · pp. 8–9

2) Agentic amplification:

  • Tool use is the most common amplification factor (4,087 listings), followed by non-determinism (1,814), autonomy (1,501), and memory use (1,229).Rarer factors include dynamic identity (560), multi-agent interaction (469), and self-modification (180).Source: 2) Agentic amplification · p. 9

3) Observed runtime activity:

Runtime traces capture only a subset of bundle capability, while dependency and declaration analyses expose broader supply-chain and permission-surface gaps. Most declarations understate capability, with error direction varying by capability class and marketplace.

3) Observed runtime activity:

  • Half the corpus can reach a shell, but only about one in twelve listings was observed invoking one during a single sandbox session.Runtime attribution is a lower bound because one session exercises few of a bundle’s code paths.Source: 3) Observed runtime activity · p. 9

4) Supply-chain surface:

  • 2,830 of 2,871 listings declaring dependencies reference at least one unpinned dependency, and 75 resolve one to a takeover-prone host.The 99% unpinned rate is calculated within the dependency-declaring population.Source: 4) Supply-chain surface · p. 9

5) Declared against actual capability:

E. Static Analysis Against Detonation

Static analysis reads bundle contents and declarations, whereas detonation records behavior exercised in a sandbox session; their findings therefore expose different aspects of risk. The phases disagree substantially, and their pattern-count differences do not establish relative sensitivity.

1) How often the phases agree:

  • A detonation-only flag is tied to a listing’s trace, but executor behavior cannot be separated from behavior driven by the skill.One session also exercises few code paths, so the count is a localized observation rather than a clean bound in either direction.Source: 1) How often the phases agree · p. 10
  • On the threat axis, either phase flags 336 listings and both flag 11; static analysis accounts for 303 and detonation for 44.Detonation reports only three threat patterns, two already known to produce false positives.Source: 1) How often the phases agree · p. 10
  • 2,361 of the 3,339 listings flagged on the risk axis were found by only one phase: 1,653 by static analysis alone and 708 by detonation alone.The table’s risk column includes any finding from either phase, including some listings below the exposure threshold used elsewhere.Source: 1) How often the phases agree · p. 10, Table III · p. 10

2) The phases see different kinds of things:

  • Detonation observes behavior, while static analysis reads declared dependencies and remit; an unexercised unpinned dependency can be visible only to static analysis.A destination reached only under a particular input illustrates the reverse case.Source: 2) The phases see different kinds of… · p. 10
  • Detonation reports compromise spreading beyond its component on 730 listings versus 526 from static analysis, while static analysis reports authority expansion on 776 versus 182.For unvetted third-party components, the counts are 703 from static analysis and one from detonation; these counts indicate gap direction, not size.Source: 2) The phases see different kinds of… · p. 10
  • These pattern gaps do not establish that either phase is more sensitive or that both measure one property with independent errors.The sensitive-data comparison is not like-for-like because the static count includes exfiltration-path composition without observed data movement.Source: 2) The phases see different kinds of… · p. 11

V. IMPLICATIONS FOR MARKETPLACE REVIEW

Capability profiles could give marketplace reviewers evidence about bundle authority, but they would not guarantee complete discovery or safe operation. The conclusions favor capability-based review alongside intent assessment, while future work identifies boundaries in profile evaluation, runtime coverage, and corpus interpretation.

V. IMPLICATIONS FOR MARKETPLACE REVIEW

  • An independently generated profile could list inferred capabilities, supporting evidence, and uncertainty, bound to the inspected bundle version.The proposed profile is hypothetical; the study does not implement or evaluate profile generation or attestation, and neither would guarantee discovery of every behavior.Source: §V · p. 11, Figure 9 · p. 11

VI. CONCLUSIONS

  • At the top two exposure levels, benign listings outnumber flagged listings four to one, so intent verdicts do not capture the larger exposure population.Every listing with a threat verdict also has medium exposure or above, but the reverse does not hold.Source: §VI · p. 11
  • The declaration vocabulary cannot name credential or environment access, and most existing declarations understate bundle capability.Where classes can be named, concrete capabilities are over-requested while diffuse ones are missed.Source: §VI · p. 11
  • The marketplaces’ popularity ordering tracks exposure but not threat, and fewer than a third of risk-flagged listings are seen by both analysis phases.A single-axis comparison therefore ranks marketplaces on that axis, not on their inventories.Source: §VI · p. 11
  • Static profiling can estimate a bundle’s capabilities at submission without execution, and combined credential access plus network egress identifies 397 listings for targeted review.The paper also counts 628 listings satisfying the lethal trifecta; gating on combinations could prioritize review at publication.Source: §VI · p. 11

VII. FUTURE WORK

  • A single sandbox run observes only the code paths it exercises; repeated sessions or path-targeted inputs could tighten the runtime lower bound.Source: §VII · p. 12
  • One corpus snapshot cannot establish whether marketplace threat ordering is stable over time, and catalog composition confounds curation comparisons.The proposed follow-ups are repeated marketplace observation and comparisons conditioned on listings with exposure.Source: §VII · p. 12

CONFLICT OF INTEREST

The scanner lacks external evaluation and ground truth, while access to the underlying scan materials is limited and does not permit full independent reconstruction.

CONFLICT OF INTEREST

  • No third party evaluated the Metano Labs scanner, and no external ground truth was available to validate its labels.Adjudication and reproducibility measures are the only checks described.Source: CONFLICT OF INTEREST · p. 12

DATA AVAILABILITY

  • The complete scan reports and bundles are not publicly released, and any limited dataset access depends on disclosure review and is not guaranteed.The offered aggregate tables and de-identified exposure index cannot independently reconstruct the complete scan results.Source: DATA AVAILABILITY · p. 12
  • Listing identities, per-listing threat verdicts, raw reports, and proprietary implementation materials are withheld.The paper provides its measurement design and validation procedures for interpreting aggregate findings.Source: DATA AVAILABILITY · p. 12

Related papers

Loading 2610.19264v1…