Structured Security Auditing and Robustness Enhancement for Untrusted Agent Skills

1 Introduction

• Pre-load auditing must recover structured risk evidence across package files and preserve stable labels under semantics-preserving rewrites.Packages can carry untrusted content in reference documents, script metadata, and repository context.Source: §1 · pp. 1–2
• The paper formalizes untrusted-skill auditing as a package-level decision problem involving hidden override, disguised transfer, and remote bootstrap chains.The approach is derived from finding experiments rather than training an additional black-box end-to-end classifier.Source: §1 · p. 2
• Strong remote baselines often collapse the suspicious/malicious distinction under rewrites and cross-file chain conflicts, rather than simply missing risk.Source: §1 · p. 2
• SKILLGUARD-ROBUST combines structured evidence extraction, selective semantic verification, conflict-aware chain arbitration, and anchor-consistency consolidation.These stages implement structure recovery, uncertainty-localized verification, and consistency-preserving adjudication.Source: §1 · p. 2
• The evaluation spans five views of 254–404 packages, separating main-distribution performance, frozen stress behavior, boundary-heavy behavior, and ecosystem transfer.The paper bounds its central claim to more stable package-level decisions on frozen evaluations, not a solved open-world guardrail.Source: §1 · p. 2

2 Related Work

• One line of prior work studies prompt injection, indirect injection, and tool-augmented agent attack surfaces (Liu et al., 2023; Zhan et al., 2024; Wallace et al., 2024).Source: §2 · p. 2
• A second line examines general guardrails and robust discrimination, including Llama Guard, Granite Guardian, and rewrite-aware safety classifiers (Inan et al., 2023; Chen et al., 2024; Li and Liu, 2024; Padhi et al., 2024; Shi et al., 2025; An et al., 2025; Li et al., 2026; Zhang et al., 2026).Source: §2 · p. 2
• A third line studies skills, repositories, and structured attack carriers (Schmotz et al., 2025; Liu et al., 2026b; Schmotz et al., 2026; Jia et al., 2026), while most prior settings use prompt-level moderation or flat long-context judgment.This paper instead requires package-scope evidence aggregation and label stabilization before loading.Source: §2 · p. 2

3 Task Definition and Benchmark

• A skill package is represented as file-content and file-role pairs, and the model must assign a security label before loading.Roles include skill-md, script, reference, and repo-context.Source: §3 · p. 2
• The deployment-oriented taxonomy treats benign packages as auto-loadable, suspicious packages as requiring a block or human review, and malicious packages as explicit attacks.Malicious operationally requires a decisive cross-file attack chain; suspicious covers boundary risks that do not establish clear override or transfer intent.Source: §3 · p. 2
• The threat model allows attackers to control some or all package content and build risk through hidden override, disguised transfer, or remote bootstrap chains.These include hidden priority instructions, disguised data delivery, and remote helpers presented as dependencies or initialization.Source: §3 · pp. 2–3
• SKILLGUARDBENCH contains 327 samples, and two public-ecosystem extensions add 254, for 581 package-level samples across five evaluation views of 254–404 packages.The views distinguish diagnostic held-outs, stronger frozen stress tests, and transfer to unseen public skill roots.Source: §3 · p. 3
• Attack exact consistency evaluates paired risk seeds and rewrites for label drift when semantics are preserved but surface form changes.Source: §3 · p. 3

4 Findings and Design Motivation

• On v1 alpha, full-package PEB raises overall exact / risk exact / attack exact / risk malicious recall from 0.6190 / 0.1765 / 0.1220 / 0 to 0.8027 / 0.7647 / 0.6829 / 0.6667.Qwen2.5-14B also improves from 0.6395 / 0.0588 / 0.0976 to 0.7891 / 0.5294 / 0.4390; the finding motivates cross-file, role-aware evidence extraction.Source: §4 · p. 3
• On held-out batch1, Qwen2.5-14B + full-package reaches 1.0000 flagged accuracy but 0 risk malicious recall, flattening every malicious sample into suspicious.The errors concentrate in semantically structured families F01/F02/F03, motivating targeted semantic verification of uncertain samples.Source: §4 · p. 3
• After semantic verification, remaining errors concentrate on over-promotion when remote bootstrap co-occurs with weak disguised transfer and on anchor–rewrite inconsistency.These residuals motivate chain-conflict arbitration followed by anchor-consistency recovery.Source: §4 · pp. 3–4

5 Method

• The four-stage decision chain recovers package structure, verifies uncertainty, arbitrates chain conflicts, and consolidates anchor–rewrite consistency.The stages target distinct recurring ambiguities rather than acting as interchangeable post-hoc enhancements.Source: §5 · p. 4

5.1 Stage I: Structured Evidence Extraction

• Role-weighted file evidence is aggregated with noisy-or, preserving strong signals from a critical file while accumulating weaker signals dispersed across files.This reconstructs package structure instead of reducing the package to a flat text span.Source: §5.1 · p. 4

5.2 Stage II: Uncertainty Triggering and Semantic Verification

• Selective semantic verification sends uncertain samples’ filtered evidence snippets for chain-level judgments; those judgments clarify ambiguity but do not directly determine the final label.Simple samples can follow a fast path to direct adjudication, while uncertain samples undergo verification before later repairs.Source: §5.2 · pp. 4–5, Figure 2 · p. 5

5.3 Stage III: Conflict-Aware Chain Arbitration

• Chain arbitration distinguishes malicious transfer dominated cases from remote-bootstrap cases when both chains co-occur and package-level confidence alone is insufficient.The method uses verifier rationale to help resolve which chain is decisive under the deployment policy.Source: §5.3 · p. 5

5.4 Stage IV: Anchor-Consistency Consolidation

• Anchor consolidation uses risk-seed rewrite clusters to repair cases where rewrites have stabilized semantically but the anchor still lags.It activates only under narrow local conditions; it is not global label propagation or uniform score smoothing.Source: §5.4 · pp. 5–6

• Five evaluation views span 254–404 samples, drawn from a 327-package core benchmark and two public-ecosystem extensions that bring the total pool to 581 samples.The views distinguish main-distribution performance, held-out behavior, frozen stress, boundary-heavy cases, and ecosystem transfer.Source: §6 · p. 6
• The baseline pool includes structured-evidence, general guardrail, and stronger remote discriminative systems, plus BUNDLEJUDGE for comparable large-view single-shot judging.Qwen2.5-14B + full-package is the strongest remote baseline in appendix results.Source: §6 · p. 6
• Reported metrics cover overall and flagged accuracy, risk exact accuracy and malicious recall, rewrite exact accuracy and malicious recall, and attack exact consistency.Flagged accuracy measures blocking risky samples, not correct recovery of the malicious class.Source: §6 · p. 6
• The enlarged views reuse the same scripts and default parameters, including fixed chain and package-floor thresholds and a requirement for at least two malicious rewrites for anchor consolidation.The stronger frozen-stress components are not tuned separately.Source: §6 · p. 6
• Only semantic verification adds remote cost; it is triggered selectively, while full-package Qwen and BUNDLEJUDGE reread every package end to end.The experiments also examine package context, baseline stability, family-level failures, method progression, and the need for arbitration and consolidation.Source: §6 · p. 7

7.1 RQ1: Is package-level context necessary?

• On v1 alpha, PEB full-package raises overall exact from 0.6190 to 0.8027 and risk malicious recall from 0 to 0.6667.The comparison supports the paper’s conclusion that package-level context is necessary for this task.Source: §7.1 · p. 7

7.3 RQ3: Why is the strongest remote baseline still insufficient?

• Strong single-shot judges often flag risk but still collapse suspicious and malicious labels; even structured evidence summaries leave BUNDLEJUDGE below the proposed method on malicious recovery and rewrite stability.The proposed method has a smaller collapse gap than the strongest remote baseline.Source: §7.2 · p. 7, §7.3 · pp. 7–8

7.2 RQ2: How does the strongest remote baseline behave across the large evaluation views?

• 97.30 overall exact and 98.33 risk malicious recall on All-HO; Qwen2.5-14B + full-package reaches 78.17 and 55.67, respectively.Table 1 covers five views, including held-out-style All-HO and Ext.-eco built from unseen public skill roots.Source: §7.2 · p. 7, Table 1 · p. 8

7.4 RQ4: What does each step of the method progression repair?

• The progression first verifies uncertain chains, then repairs narrower chain-dominance conflicts, and finally resolves remaining anchor–rewrite residuals.SKILLGUARD-VERIFY already brings all five large views into the mid-to-high 90s.Source: §7.4 · p. 8

7.5 RQ5: What do the ablations show?

• Later-stage gains are largest on All-HO, Int.-stress, and Boundary, where localized uncertainty has not yet become a stable package decision.The stages add focused, view-dependent improvements rather than generic smoothing.Source: §7.5 · p. 8

• 97.30 overall exact and 98.33 risk malicious recall on the held-out aggregate; the external-ecosystem view reaches 99.66 and 100.00, respectively.The reported external-ecosystem result also includes 100.00 attack exact consistency.Source: §8 · p. 8
• The paper identifies suspicious/malicious collapse, rather than generic risk blindness, as the dominant failure mode in package-level skill auditing.Its approach combines structure recovery, selective semantic verification, and consistency-preserving adjudication.Source: §8 · p. 8
• The results strengthen evidence for frozen and ecosystem-facing robustness, but harsher external-source transfer remains an open challenge.Source: §8 · p. 8

Limitations

• Risk samples partly use sanitized reconstructions, so results are better suited to method comparison than estimating real-world prevalence.The paper says a substantial portion does not come from native malicious repositories in the open ecosystem.Source: Limitations · p. 9
• Batch1 and batch2 are development-linked, so their saturated results are repaired within-suite residuals rather than fully development-free generalization.Batch3–6 offer stronger frozen stress evidence, but benchmark–method co-evolution cannot be ruled out completely.Source: Limitations · p. 9
• Suspicious-versus-malicious boundaries depend on deployment context, and preserved audit artifacts do not replace larger independent annotator-agreement studies.Some transfer cases may warrant direct-malicious labels in high-security settings but manual review in more permissive ones.Source: Limitations · p. 9
• The framework relies on one strong verifier, and harder source shifts can defeat verifier-floor estimation and consistency repair together.The paper frames the approach as an auditable path, not a final solution to open-world skill security.Source: Limitations · p. 9

B Related Work Supplement: Extended Positioning and Comparison

• Unlike interaction-time attack studies, this work asks whether a third-party, repository-like skill package should be admitted before execution.The compared work addresses agent vulnerabilities during execution or interaction.Source: B Related Work Supplement: Extended… · p. 12
• Prior guardrail work adds intermediate reasoning, while this paper audits structured packages and sends only uncertain cases to a verifier.The cited work usually treats a text span, interaction, or dialogue context as its judgment unit.Source: B Related Work Supplement: Extended… · p. 12
• Structural safety work motivates attention to relations among file roles, evidence chains, risk seeds, and rewrites rather than isolated dangerous phrases.The supplement connects package auditing to work on instruction hierarchy, structural changes, and benign-looking attack carriers.Source: B Related Work Supplement: Extended… · p. 12
• Where prior skills research demonstrates attack carriers or classifies skills, this paper studies pre-load auditing through evidence aggregation, local verification, and consistency recovery.The appendix roadmap maps supplementary materials to the paper's related work, benchmark, findings, method, experiments, results, external validity, and reproducibility.Source: B Related Work Supplement: Extended… · p. 12, A Appendix Roadmap · p. 12

Large evaluation views

• The benchmark recombines held-out splits with two public-ecosystem extensions into five large evaluation views—Main, All-HO, Int.-stress, Boundary, and Ext.eco—separating robustness questions at a scale where coarse 1/30 jumps no longer dominate.Source: C Finding-Experiment Supplement:… · p. 13

Input-view comparison

• Switching from SKILL.md-only to full-package input improves both PEB and Qwen2.5-14B in the same direction, indicating auditing fails when the package is modeled as a single-file prompt rather than a structured multi-file object.Source: Figure 4 · p. 13, C Finding-Experiment Supplement:… · p. 13

Directional failure patterns

• Across large views, the strongest remote baseline flags risky packages but under-recovers malicious intent; staged methods nearly eliminate the resulting collapse gap.The structured single-shot control improves on the flat judge but retains a visible gap.Source: C Finding-Experiment Supplement:… · pp. 13–14, Figure 5 · p. 14

Method progression

• The gain from Qwen to SKILLGUARD-VERIFY is already large across four large views, while later stages bring decisions near saturation by repairing narrower residuals.SKILLGUARD-CALIBRATE addresses chain-dominance conflicts; SKILLGUARD-ROBUST removes remaining anchor–rewrite residuals.Source: C Finding-Experiment Supplement:… · p. 14

Ablation figure

• SKILLGUARD-CALIBRATE adds the largest overall-exact gains on All-HO and Int.-stress, while SKILLGUARD-ROBUST makes smaller consistency gains on anchor–rewrite residuals.The stages address decreasing scopes rather than stacking undifferentiated heuristics.Source: C Finding-Experiment Supplement:… · p. 14

D Experimental-Setup Supplement: Full Baseline Results

• Across Main, All-HO, Int.-stress, Boundary, and Ext.-eco, semantic verification provides most of the gain before later stages convert narrower residuals into near-saturated decisions.Figure 6 presents the main results and method progression across the large evaluation views.Source: Figure 6 · p. 15
• SKILLGUARD-CALIBRATE contributes larger medium-scope repairs, while SKILLGUARD-ROBUST removes remaining consistency residuals across the large-view progression.Figure 7 summarizes the stage-wise gains.Source: Figure 7 · p. 15
• Granite Guardian tends to over-defend, Llama Guard 3 under-defends, Llama Guard 4 is more balanced but rarely recovers malicious cases, and Qwen2.5 models remain susceptible to family-level drift.The fuller v1 alpha baseline table and large-view table provide the comparison scopes, but these passages do not report their cell values.Source: D Experimental-Setup Supplement: Full… · p. 15, Table 2 · p. 16, Table 3 · p. 16
• 85.13 / 90.11 / 93.72 overall exact on Int.-stress, Boundary, and Ext.-eco leaves BUNDLEJUDGE below the factorized family on malicious recall and rewrite stability.The comparison uses structured evidence and selected files, making it a stronger control than a flattened full-package judge.Source: D Experimental-Setup Supplement: Full… · pp. 15–16

E Task and Benchmark Supplement: Data Construction, Annotation, and Sanitization

• 581 package-level samples comprise the expanded pool, with five complementary views ranging from 254 to 404 packages; rewrites preserve attack intent while changing style, evidence location, and cover story.The views include Main, All-HO, Int.-stress, Boundary, and Ext.-eco.Source: E Task and Benchmark Supplement: Data… · pp. 16–18

H Results and Discussion Supplement: Large-View Diagnostics

• 37.35 on All-HO and 32.74 on Int.-stress are Qwen2.5-14B flagged-accuracy minus malicious-recall gaps; SKILLGUARD-ROBUST nearly eliminates the gap across all five large views.The same diagnostics report smaller BUNDLEJUDGE gaps on Int.-stress, Boundary, and Ext.-eco.Source: Table 4 · p. 17, H Results and Discussion Supplement:… · p. 20
• BUNDLEJUDGE also retains collapse gaps, while later-stage gains are smaller than the semantic-verification jump and target chain conflicts and anchor–rewrite inconsistency.Public-ecosystem views extend the evaluation to unseen public roots and open-source attack carriers rather than replacing the other views.Source: Table 4 · p. 17, H Results and Discussion Supplement:… · p. 20

E Task and Benchmark Supplement: Data Construction, Annotation, and Sanitization

• Malicious labels require decisive cross-file attack evidence, while suspicious covers review-worthy boundary risks; a 60-package blinded boundary pack is a reproducibility asset, not additional statistical evidence for the main claims.The pack contains 30 suspicious and 30 malicious cases across 11 boundary families.Source: E Task and Benchmark Supplement: Data… · p. 17

F Results Supplement: Additional Analyses and Cases

• Reference-only overrides, disguised transfers, bootstrap-transfer conflicts, and ordinary repository scaffolding expose distinct cases where evidence aggregation and semantic verification help distinguish malicious intent from suspicious risk.The public-ecosystem results report 100.00 risk malicious recall for SKILLGUARD-ROBUST, versus 83.00 for Qwen2.5-14B and 88.30 for BUNDLEJUDGE on Ext.-eco.Source: F Results Supplement: Additional… · pp. 18–19

G Method Supplement: Implementation Details and Trigger Statistics

• Structured extraction runs on every sample, but semantic verification, chain arbitration, and anchor consolidation trigger selectively; later stages address narrower residuals rather than rereading every package.From VERIFY to CALIBRATE, overall exact rises 4.23 points on All-HO and 3.45 on Int.-stress; subsequent gains are narrower.Source: G Method Supplement: Implementation… · p. 19
• The large views reuse fixed defaults after progression on earlier batches, so they serve as scaled transfer checks rather than separately tuned evaluation settings.The method is framed as a factorized decision chain because stronger flat and structured single-shot judges retain malicious-recovery and rewrite-stability gaps.Source: G Method Supplement: Implementation… · pp. 19–20

I Reproducibility and Discussion Supplement

• The near-saturated results are not proof that every open-world variation is solved, and the large views combine the core benchmark with public-ecosystem extensions.Batch1 and batch2 participated in the original method progression before defaults were reused on the larger views.Source: I Reproducibility and Discussion… · p. 21