SkillDRE: Dual-Stage Red-Team Evolution of Agent Skills via Pre-Execution and Runtime Feedback

1 INTRODUCTION

• Attack construction must reach the harmful behavior during a legitimate workflow, execute successfully, and survive defenses inspecting both the package and runtime operations.A scan-passing candidate may miss its target, while a revision that repairs execution may introduce scanner findings.Source: §1 · pp. 1–2
• Prior automated methods refine instructions or code using execution traces, detector findings, or audit feedback, motivating evolution of the complete package across both defense stages.Optimizing either objective in isolation can undo progress on the other.Source: §1 · p. 2
• SkillDRE fixes a task-conditioned malicious objective and verifiable judge rule, then evolves the skill implementation using scanner and runtime feedback in a cross-stage loop.Runtime-guided revisions are rescanned before further execution.Source: §1 · p. 2
• 45.28% average attack success rate (ASR), exceeding the strongest baseline by 40.3%, with 0% SkillScan detection and largely preserved benign-task performance.The evaluation covers 249 Skills associated with 94 SkillsBench tasks across four victim models.Source: §1 · p. 2
• SkillDRE’s threat model measures legitimate task functionality separately while the attacker revises a skill using scanner findings and runtime-defense decisions.The task, victim agent, and defenses remain fixed.Source: §1 · p. 2

2.1 AGENT SKILLS AND SELF-EVOLUTION

• Most skill-evolution research focuses on improving benign task capability through skill acquisition, refinement, transfer, composition, or iterative improvement in persistent libraries.The literature also studies transfer across tasks, models, or execution systems.Source: §2.1 · pp. 2–3

2.2 ATTACKS ON AGENT SKILLS

• Attack methods modify prompts or skill packages and use varied feedback, whereas SkillDRE fixes a task-conditioned target and judge rule and returns runtime-guided revisions for rescanning.The comparison covers modified content, target construction, and refinement feedback.Source: §2.2 · p. 3

3 METHODOLOGY

• The framework alternates scanner-guided pre-execution refinement with runtime-guided refinement, routing each runtime revision back through pre-execution scanning.The attack target and judge rule remain fixed throughout the loop.Source: §3 · p. 3, Figure 1 · p. 4

3.1 PRELIMINARIES AND THREAT MODEL

• The attacker modifies only the target skill while the task, victim agent, other skills, and defenses remain fixed across repeated trials.A victim agent executes the benign task with its associated skills in a sandbox.Source: §3.1 · pp. 3–4

3.2 INITIALIZATION: ATTACK TARGET AND JUDGE RULE CONSTRUCTION

• Initialization constructs a shared task-level intent and per-skill targets, then builds judge rules using local checks and semantic validation.Accepted targets and the final judge rule remain fixed during skill evolution.Source: §3.2 · pp. 4–5

3.3 PHASE 1: PRE-EXECUTION EVOLUTION

• Phase 1 scores SkillScan findings by severity and admits only zero-risk candidates to Phase 2, while refiners preserve the skill’s functionality and interface.Lower risk scores indicate lower scanner-assessed risk.Source: §3.3 · pp. 5–6

3.4 PHASE 2: RUNTIME-GUIDED EVOLUTION

• Phase 2 uses execution and defense feedback to seek target realization without an operation stop; each revision re-enters Phase 1 for rescanning.The judge rule’s diagnostics and SkillSonar’s decisions guide runtime refinement.Source: §3.4 · p. 6

4.1 EXPERIMENTAL SETUP

• The evaluation covers 249 Skills and 94 tasks, uses four victim models, and reports ASR under runtime defense, SkillScan detection rate, and legitimate-task accuracy.All attack methods use DeepSeek-V4-Pro as the attack model; Phase 2 allows at most ten runtime trials per skill.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

• 45.28% average ASR across four victim models, 40.3 percentage points above the strongest baseline; SkillDRE has 0.00% DR versus baselines’ 97.34–100.00%.Per-model ASR ranges from 43.78% to 46.99%.Source: Table 2 · p. 7, §4.2 · p. 7
• Figure 2 pairs No Attack and SkillDRE benign-task ACC within each victim model; ∆ACC is measured in percentage points, with positive values indicating degradation.Source: Figure 2 · p. 7

4.3 CONSTRUCTION EFFICIENCY AND EVOLUTION DYNAMICS

• Task-level intents, skill-level targets, and judge rules take averages of 1.49, 2.24, and 1.84 construction rounds, with first-round acceptance rates of 69.15%, 80.72%, and 61.04%, respectively.Accepted targets and judge rules remain fixed during skill evolution; skill-level targets have the highest first-round acceptance but extend to 29 rounds.Source: §4.3 · p. 8, Figure 3 · p. 8
• Cold-start evolution supplies scanner-approved skills for Phase 2; Figure 4 encodes rounds to scanner acceptance and aggregate findings by severity.Source: Figure 4 · p. 8, §4.3 · p. 9
• 34.54–37.35% first-round ASR rises to 43.78–46.99% after ten runtime rounds, a gain of 8.03–12.45 percentage points across victim models.Every revised skill must again pass SkillScan before execution; by round 8, all four models have achieved most observed gains.Source: §4.3 · p. 9, Figure 5 · p. 9

4.4 ABLATION RESULTS

• 46.99% ASR with 0.00% DR for full SkillDRE exceeds iterative Phase 1 by 12.05 ASR points and reduces DR by 79.92 points versus iterative Phase 2.Iterative Phase 1 alone lowers DR but also lowers ASR; iterative Phase 2 raises ASR while increasing DR.Source: §4.4 · p. 9, Table 3 · p. 9

5 CONCLUSION

• 45.28% average ASR across four victim models, exceeding the strongest baseline by 40.3%, with no SkillScan risk findings on final submitted skills and largely preserved benign-task performance.Source: §5 · p. 10

A OVERALL ALGORITHM OF SKILLDRE

• SkillDRE constructs and validates an attack target and judge rule, holds both fixed, then combines scanner-guided evolution with runtime-guided refinement and rescans revisions before execution.The fixed objective separates the targeted harm from the implementation used to realize it.Source: §5 · p. 10, A OVERALL ALGORITHM OF SKILLDRE · pp. 14–15

5 CONCLUSION

• Ablations show runtime-guided refinement increases attack success, while Phase 1 re-entry restores scanner acceptance after runtime-guided edits.Source: §5 · p. 10

• The score R(h, m, l) = w_hh + w_mm + w_ll encodes lexicographic minimization of high-, then medium-, then low-severity findings; this is a ranking choice, not calibrated relative harm.Source: B RATIONALE FOR RISK-SCORE WEIGHTS · p. 15
• For bounded counts m ≤ M and l ≤ L, w_h > w_mM + w_lL, w_m > w_lL, and w_l > 0 suffice to preserve the severity ordering.An extra higher-severity finding cannot be offset by reductions in lower-severity counts.Source: B RATIONALE FOR RISK-SCORE WEIGHTS · p. 15
• Across 804 valid scans, observed M = 9 and L = 5; weights (10,000, 1,000, 10) satisfy the stated ordering conditions.Within this count range, minimizing the weighted score is equivalent to lexicographically minimizing the severity-count triplet, and R = 0 only when all counts are zero.Source: B RATIONALE FOR RISK-SCORE WEIGHTS · p. 15
• Among 1,703 candidate pairs, adopted weights produced no inversions or ties, versus 107 inversions and 303 ties for equal weights and 13 inversions and 66 ties for (3, 2, 1).Comparisons were restricted to distinct severity-count triplets from the same task–skill combination.Source: B RATIONALE FOR RISK-SCORE WEIGHTS · p. 15

• ∆ACC = ACC_no attack − ACC_attack is computed using original SkillsBench evaluators; positive values indicate accuracy degradation.Source: C COMPARATIVE ANALYSIS OF ATTACK… · p. 15, Table 4 · p. 16
• SkillHarm’s ∆ACC is computed on its filtered attack-evaluated subset, with no-attack and attack evaluations using the same Skills; other methods use the full benchmark.Source: C COMPARATIVE ANALYSIS OF ATTACK… · p. 15
• SkillDRE averages −2.12 percentage points ∆ACC across four victim models, compared with 9.13 for SkillJect and 6.59 for SkillHarm.SkillDRE has negative ∆ACC on three models and a marginal positive change of 0.15 points on GLM-5.2.Source: C COMPARATIVE ANALYSIS OF ATTACK… · p. 15, Table 4 · p. 16

D THREAT CLASSIFICATION STATISTICS

• Exfiltration-related behaviors account for 72.29% of classified threats.These include File & Media, Business Data, and Source Code Exfiltration.Source: D THREAT CLASSIFICATION STATISTICS · p. 16
• File & Media Exfiltration is the largest category at 34.94%, followed by Business Data Exfiltration at 33.33%.Source: D THREAT CLASSIFICATION STATISTICS · p. 16
• Privacy Leakage accounts for 10.84% of threats, Backdoor Attacks for 8.84%, Result Manipulation for 8.03%, and Source Code Exfiltration for 4.02%.Source: D THREAT CLASSIFICATION STATISTICS · p. 16

E HUMAN VALIDATION OF ATTACK TARGETS

• Reviewers uniformly sampled 125 of 249 skill-level instances without replacement and checked target maliciousness and judge-rule decisions on actual execution trajectories.Source: E HUMAN VALIDATION OF ATTACK TARGETS · p. 16
• Human annotators agreed with the model validator on target maliciousness in 100% of reviewed cases.All reviewed targets were judged to specify behavior beyond legitimate task requirements.Source: E HUMAN VALIDATION OF ATTACK TARGETS · p. 16
• Human reviewers agreed with model-based judge-rule validation in 96.8% of reviewed cases.Source: E HUMAN VALIDATION OF ATTACK TARGETS · p. 16

F THREAT CATEGORY DISTRIBUTION IN PHASE 1

• The analysis retains SkillScan’s original High, Medium, and Low risk labels without manual relabeling or post-hoc normalization.The labels indicate relative risk, and the same threat category can appear at different risk levels.Source: F THREAT CATEGORY DISTRIBUTION IN… · p. 16
• Data Exfiltration accounted for 23.65% of all Phase 1 findings, Unauthorized Tool Use for 23.59%, and Skill Discovery Abuse for 17.53%.Source: Table 5 · p. 17, F THREAT CATEGORY DISTRIBUTION IN… · p. 17
• High-risk findings were concentrated in Data Exfiltration at 26.81% and Command Injection at 23.55%, while medium-risk findings primarily involved Unauthorized Tool Use at 30.14% and Data Exfiltration at 28.87%.Source: F THREAT CATEGORY DISTRIBUTION IN… · p. 17, Table 5 · p. 17
• Low-risk findings were more distributed: Unauthorized Tool Use accounted for 23.06%, Skill Discovery Abuse 22.65%, Data Exfiltration 18.10%, and Harmful Content 16.22%.Source: F THREAT CATEGORY DISTRIBUTION IN… · p. 17, Table 5 · p. 17

G HARNESS COMPARISON

• SkillScan DR measures detection on final submitted candidate skills, not runtime detection by either harness.Source: G HARNESS COMPARISON · p. 17
• The comparison used DeepSeek-V4-Pro as the victim model in both harness settings.Source: G HARNESS COMPARISON · p. 17
• 46.99% Codex ASR and 42.57% Claude Code ASR were reported, while SkillScan DR remained 0% in both harnesses.The results indicate effectiveness under both tested harnesses, not only Codex.Source: Table 6 · p. 17, G HARNESS COMPARISON · p. 17