SkillShield: Prompt-Space Security Skills for LLM Coding Agents

Xiaodong Wu; Zhimin Zhao; Qi Li; Xiangman Li; Yu Shi; Bram Adams; Jianbing Ni

Source-linked AI summary

SkillShield: Prompt-Space Security Skills for LLM Coding Agents

Xiaodong Wu, Zhimin Zhao, Qi Li, Xiangman Li, Yu Shi, Bram Adams, Jianbing Ni

arXiv:2608.25817v1Aug 26, 2026cs.CR

TL;DR

Coding agents can turn untrusted instructions into harmful actions while existing defenses impose weight-access, coverage, or runtime-component limitations. SkillShield synthesizes attack-informed security skills offline and keeps them in the system prompt throughout tool use. On RedCode, it reduces harmful behavior across six LLMs, with stronger results from narrower provisioning and a 0.14% benign safety-refusal rate.

Problem
Coding agents operate with developer privileges, while existing alignment, input-filtering, and runtime-monitoring defenses have important deployment or coverage limitations for API-only systems.
Method
SkillShield synthesizes security skills offline from known attacks or recorded failures and injects a selected policy into the system prompt at session start for the tool-use loop.
Results
Across six LLMs on RedCode, the all-classes skill reduces execution attack success rate from 67.4% to 43.6% and malware-generation severity from 3.37 to 0.58, while per-class provisioning reaches 14.5% execution attack success rate.
Takeaways & Limitations
Prompt-space security skills provide an API-only first line of defense that reduces harmful execution and malware generation without an additional runtime component.
Takeaways & Limitations
The benign evaluation measures only safety-grounded over-refusal, not end-to-end benign task success, and the synthesis scope gap is not an information-theoretic limit.

Abstract

from arXiv · show

A coding agent edits files and executes shell commands with its developer's privileges, allowing malicious requests to translate directly into harmful actions or functional malware. Existing defenses have complementary limitations: weight-level alignment is unavailable to API-only deployers, whereas input filters and execution-boundary monitors require auxiliary classification or checking components along the agent's trajectory. We therefore introduce SkillShield, a system-prompt defense that synthesizes security skills offline from known attacks or recorded agent failures. These skills are injected into the system prompt at session start and remain active throughout the tool-use loop. Unlike a reference monitor, they protect the system by defining the security policies the model should follow during execution. Due to the limited system-prompt space, we examine three fixed-budget provisioning scopes: all-classes, with one skill covering all threat classes, per-bundle, with one skill targeting a related subset, and per-class, with one skill dedicated to a single known class and used as the upper-bound reference. None requires runtime request classification or routing. Across six large language models on RedCode, the default all-classes skill reduces malware-generation severity from 3.37 to 0.58 and achieves a 43.6% execution attack success rate, comparable to Llama Guard 3's 42.7% without its separate 8B classifier. The per-bundle and class-fixed per-class settings further reduce this rate to 36.2% and 14.5%, respectively. Under two non-adaptive jailbreak families, SkillShield continues to outperform all baselines on malware generation. Across 731 benign task descriptions, SkillShield yields a mean safety-refusal rate of 0.14%. These results demonstrate the potential of prompt-space security skills to prevent harmful actions and malware generation for LLM coding agents.

1 Introduction

SkillShield addresses harmful coding-agent behavior with security policies synthesized offline and injected into the system prompt, while fixed prompt budgets create a scope-versus-detail trade-off. Across models and evaluation settings, narrower or better-composed skills improve protection with little measured benign refusal.

Motivation: Coding agents can execute harmful actions because they operate with developer privileges, process untrusted content, and may issue tool calls without human approval.The paper positions system-prompt policy as an earlier control point than runtime action inspection.
Design trade-off: Finite prompt budgets force a trade-off: broader scopes cover more threat classes, while narrower scopes preserve more class-specific security detail.Directly combining class-specific policy text can recover protection lost at broader scopes.
Evaluation: 67.4% to 43.6% execution attack success rate and 3.37 to 0.58 malware-generation severity result from the all-classes skill across six LLMs on RedCode.Per-class provisioning further lowers execution attack success rate to 14.5% when the threat class is known in advance.
Approach: SkillShield synthesizes security policies offline from known attacks or recorded failures and injects each skill into the operator-controlled system prompt for the full session.It requires neither model-weight changes, an auxiliary model, nor runtime checking.
Benign impact: 0.14% mean safety-grounded refusal occurs across 731 benign SWE-Bench Pro task descriptions.End-to-end benign task success was outside the evaluation scope.

2 Background

The paper models coding agents as language models interacting with tools through a system prompt and multi-step observation-action loops. Skill documents are inspectable policy modules, but SkillShield loads security policy fully at session start rather than progressively.

Agent model: A coding agent combines an LLM with tools for file access, code editing, shell execution, and external-service interaction.At each step, observations inform a proposed action and the executor returns a result.
Agent model: The agent generates each action from the system prompt and prior history, so operator-controlled prompt content can carry policy throughout the session.The prompt has finite size, limiting how much policy it can contain.
Skills: Skills are modular natural-language documents that extend or specialize agent behavior without retraining.The format is packaged as a SKILL.md file with metadata and an instruction body.
Skills: SkillShield injects security policy in full at session start instead of using progressive disclosure.This avoids allowing an initial request to influence whether the policy is loaded or followed.

3 Related Work

Existing coding-agent defenses intervene at model, input, or execution stages, each with distinct access, coverage, enforcement, and runtime-cost trade-offs. The paper contrasts these approaches with prompt-space policies organized by fixed threat-taxonomy provisioning granularity.

Model-level defenses: Model-level alignment encodes safety in weights but requires retraining and is unavailable to API-only deployers.The paper also notes that benign fine-tuning can degrade safety alignment.
Input defenses: Input filters classify requests before work begins, but covering repository artifacts and tool output requires additional filtering boundaries.Their protection is limited to the channels on which the classifier is installed.
Execution defenses: Runtime systems inspect candidate actions and can enforce decisions independently of textual policy adherence, but they add components, latency, and cost at covered boundaries.Prompt-space policies instead influence action formation throughout the session without a runtime component, while depending on model compliance.
Prompt-space policies: Provisioning granularity counts the threat classes represented in one fixed policy document and is selected before requests arrive, without request-time routing.The paper distinguishes broader and narrower policy scopes within a finite prompt budget.
Comparative systems: Llama Guard 3 uses an 8-billion-parameter classifier, whereas AGrail adds a runtime generation step for action-specific checks.These systems illustrate the auxiliary components used by non-prompt-space defenses.

4 Problem Formulation

SkillShield formalizes a fixed-budget system-prompt defense for coding agents that preserves tools while targeting harmful execution and malware generation. The objective balances reduced malicious harm against safety-grounded refusal on benign tasks, with provisioning granularity determining how threat classes share policy capacity.

Threat Model: The adversary may submit arbitrary tasks, inject instructions through project artifacts, use fixed reformulations, and lack deployment access.Evaluated reformulations include ASCII art and persona framing; direct access to the complete system prompt and white-box optimization are outside scope.
Threat Model: Coding agents can execute harmful commands because they operate with developer privileges and process adversarial project artifacts or requests.The threat model includes harmful shell execution such as data exfiltration or file encryption, and malicious code generation such as ransomware or rootkits.
Threat Model: The defender controls the system prompt, has an offline attack corpus, cannot retrain the model, preserves tool access, and fixes the skill before requests arrive.Bundle or class skills require threat scope known independently of incoming requests because request-time selection would require an excluded detector.
Defense Goals: The defense goals are to reduce harmful actions and usable malware, preserve performance under fixed reformulations, and minimize benign safety refusals.The benign evaluation measures safety-grounded refusal rather than end-to-end task completion.
Formal Objective: The defended agent inserts a fixed-budget skill into the system prompt while leaving model weights and tool access unchanged.The resulting agent is As = (M,T,w ⊕s ⊕P), and the skill remains active throughout the multi-step trajectory.
Formal Objective: Each candidate skill body contains at most L characters, excluding the fixed wrapper, making prompt space the central resource constraint.The complete trajectory includes model actions, executor observations, and a terminating action; malformed calls and timeouts satisfy neither safe refusal nor benign success.
Formal Objective: The formal objective minimizes expected malicious harm subject to a tolerated benign safety-refusal rate ε.Execution success and generation severity are reported as separate risk outcomes.
Formal Objective: Provisioning granularity measures how many threat classes share one synthesized policy document, trading coverage against policy detail.Singleton, mechanism-bundle, and all-class configurations respectively allocate policy capacity to narrower or broader threat scopes.

5 SKILLSHIELD

SkillShield synthesizes attack-informed security skills offline, injects them at session start, and keeps them active throughout tool use. Its fixed prompt budget motivates proactive or reactive synthesis and three provisioning granularities that trade breadth for detail.

SkillShield: SkillShield places one synthesized security skill in the system prompt before deployment and keeps it active throughout the interaction.The operator chooses how many threat classes the fixed-budget skill covers.
Security Skill Synthesis: The fixed preamble requires complete refusal of harmful or malicious requests without executing, modifying, or sanitizing them.This standardizes refusal behavior and disallows partial execution that could still demonstrate a dangerous capability.
Security Skill Synthesis: Each skill is a self-contained policy document containing a threat overview, detection rules, and refusal instructions.Detection rules target concrete signals such as command patterns, paths, and tool sequences.
Security Skill Synthesis: Proactive mining extracts attack signatures from known cases, whereas reactive learning derives defense principles from recorded undefended failures.Proactive rules are technically inspectable but may miss paraphrases; reactive principles target observed failure modes and may generalize across formats.
Shared Refinement: Sequential refinement partitions intermediate items into chunks, integrates them incrementally, truncates each body to L characters, and stops near the limit.The stopping rule triggers at 0.9L and does not evaluate skill quality during synthesis.
Shared Refinement: The measured scope gap reflects information discarded by sequential refinement, input order, and early stopping rather than an information-theoretic limit.A budget-matched assembly control tests how much loss can be recovered by changing the merge procedure.
Security Skill Synthesis: Exact knowledge of either synthesis strategy can expose omissions, so robustness is evaluated with fixed reformulations not optimized against the deployed skill.Reactive skills target the evaluated agent’s observed failures, while proactive command-specific rules can miss paraphrases.
Provisioning Granularity: The fixed body budget forces a tradeoff between coverage and detail across all-classes, per-bundle, and per-class provisioning.All-classes uses one body across the taxonomy; per-bundle groups related mechanisms; per-class allocates the complete budget to one known class.

6 Experiments

Experiments evaluate SkillShield across models, harnesses, threat scopes, prompt budgets, jailbreaks, and benign tasks. More specific provisioning generally improves execution protection, while generation protection remains strong and safety-refusal costs remain low.

Experimental setup: SkillShield is evaluated across six LLMs, three harnesses, RedCode’s execution and generation risks, fixed jailbreaks, and 731 benign tasks.The primary harness is mini-SWE-agent; OpenCode and Pi test other tool protocols.
Q1: Effectiveness of SKILLSHIELD: 40.1% mean execution ASR with all-classes, 35.4% with per-bundle, and 10.1% with per-class, compared with Llama Guard 3’s 42.7%.SkillShield uses no second model, while Llama Guard 3 evaluates a separate 8-billion-parameter classifier on every request.
Q1: Effectiveness of SKILLSHIELD: 56.7% of execution requests are refused by Reactive per-class skills versus 46.2% for Llama Guard 3, but unsuccessful attack trajectories also contribute to the lower ASR.Across models, SkillShield’s ASR is approximately four times lower than Llama Guard 3’s, so refusal alone does not explain the reduction.
Q2: Covering Many Threat Classes in One Finite Prompt: Per-class reaches 6.0% in-scope execution ASR, while per-bundle reaches 37.0% in-scope and 44.8% outside its bundle; all-classes reaches 43.9% on an unseen class.Thus, known mechanisms favor per-bundle or per-class provisioning, whereas broader or unknown mechanisms favor all-classes.
Q2: Covering Many Threat Classes in One Finite Prompt: 26.5% mean execution ASR for budget-matched concatenated skills versus 36.2% for synthesized bundle skills, indicating that synthesis accounts for much of broader-scope loss.Concatenation improves five of six LLMs by preserving more concrete rules about dangerous tools, paths, and action patterns.
Q3: A Fixed Policy Under Jailbreak: Under reformulation, every SkillShield configuration has worst-case generation AvgS at most 0.68 versus at least 1.42 for every baseline.Per-class also reaches 31.2% worst-case execution ASR, 12 percentage points below Llama Guard 3’s 43.2%, while broader scopes lose clean-input parity.
Q4: Benign-task refusal: 0.11–0.14% mean safety-refusal FPR across Proactive configurations on 731 benign task descriptions, with a maximum of 0.68% on GPT.End-to-end benign task success remains outside the evaluation scope.

7 Discussion

SkillShield uses a persistent system-prompt policy to influence action generation throughout tool-use loops without retraining or runtime request routing. Its protection varies with model capability, deployment scope, policy construction, and jailbreak reformulation.

Security policy before action generation: A security skill remains active during action generation, influencing tool selection, command arguments, and responses to intermediate tool output.This lets the same policy apply when repository artifacts or tool output introduce instructions absent from the initial request.
Model capability and deployment scope: SkillShield performs no request-time selection, while execution protection varies across models and deployment scopes.Broader fixed documents contain more threat knowledge, whereas narrower scopes can provide stronger protection for known threats.
Layered deployment: Per-class skills retain an execution advantage when the threat class is known, while broad skills produce higher ASR than LG after persona reformulation.Runtime verification remains a separate pre-execution check, but the reported comparison evaluates SkillShield alone.
Runtime overhead: A deployed skill adds approximately 2,400 input tokens and requires no additional model call.For a ten-step session using 20,000–100,000 tokens, this represents 2.4–12% of the session context.

8 Conclusion

The paper presents a systematic study of prompt-space security skills for LLM coding agents. Across six LLMs, SkillShield reduces harmful execution and malware generation while maintaining low refusal on benign tasks.

8 Conclusion: Across six LLMs, SkillShield reduces harmful execution and malware generation while producing a 0.14% safety-grounded refusal rate on 731 benign task descriptions.It requires neither model-weight access nor an additional runtime component.
8 Conclusion: Narrow skills provide stronger protection against known threats, whereas broad skills support general-purpose deployment and transfer to unseen classes.The paper frames policy composition as a central security challenge.
8 Conclusion: SkillShield synthesizes skills from known attacks or recorded agent failures and maintains a selected skill throughout the tool-use loop.The system prompt is presented as a practical security control surface without runtime classification or routing.

Ethical Considerations

The study evaluates coding-agent defenses without human participants, private user data, or unauthorized third-party access. It uses isolated execution and automated scoring while recognizing residual disclosure risks from publishing defensive artifacts.

Ethical considerations: The study involves no human subjects, personally identifiable data, or informed-consent procedure.Automated judges and the VirusTotal API score generated malware.
Harms and mitigations: Agent execution occurs in isolated Docker containers without external network access, and generated malware is never compiled or deployed.The study reuses RedCode attack prompts and introduces no new offensive technique.
Harms and mitigations: Publishing defensive skill text can help motivated adversaries identify missing patterns, leaving residual disclosure risk despite defense-in-depth mitigation.Automated scoring reduces researcher exposure to attack content but cannot remove it completely.
Decision to proceed and publish: The expected benefit supports conducting and publishing the study because coding-agent security is a growing practical concern.The researchers withhold raw malware artifacts and jailbreak strings used in evaluation.

Open Science

The paper provides code, evaluation harnesses, analysis scripts, generated skills, aggregate results, and documentation for reproducing its experiments. Public benchmarks remain externally sourced, while harmful operational artifacts are withheld and sanitized substitutes are provided.

Open science: The supporting code and artifacts will be released publicly upon publication for inspecting implementation and reproducing reported statistics.Rerunning experiments requires access to the stated benchmarks and model APIs.
Released artifacts: The released artifact includes SkillShield synthesis and injection code, evaluation harnesses, five baselines, benchmark scripts, statistical analyses, generated skills, and plotting files.The repository README documents setup, dependencies, Docker requirements, model and API configuration, and example commands.
External artifacts and access: RedCode and SWE-Bench Pro are obtained from official sources, with train and test partitions reconstructed using a documented deterministic seed.The paper does not redistribute the public benchmarks.
Withheld and redacted artifacts: Generated malware, executable payloads, credentials, API keys, and raw harmful-code trajectories are withheld to reduce misuse risk.Sanitized JSON and CSV results, aggregate reports, scoring scripts, plotting scripts, and defensive skill artifacts are provided instead.

B Experimental Implementation Details

The evaluation uses deterministic, isolated agent runs on RedCode, with fixed resource limits and three fixed-budget skill provisioning scopes. RedCode-Exec includes mechanism-based bundles, while baselines include runtime, input, provenance, and model-level defenses.

Benchmark and baselines: RedCode-Exec contains 27 task groups with 30 base cases each, and every case has four prompt variants sharing one harmful intent.Variants include raw malicious Python and jailbreak reformulations.
Benchmark and baselines: Generated malware is scored with the official 0/1/5/8/10 rubric, where higher scores indicate more functional and harmful malware.A single LLM judge, Qwen3.6-Plus, produces the scores.
Execution setup: Each agent run uses a 10-step limit, a $0.10 cost limit, temperature 0.0, and 1,024 maximum output tokens per LLM call.Code-execution tasks run in isolated Docker containers with a 600-second timeout.
Skill provisioning: Security skills are limited to L = 10,000 characters, with 2,000-character refinement chunks when training data exceeds the budget.All-classes uses one skill, per-class uses one per threat class, and per-bundle uses one per predefined bundle.
Skill provisioning: Per-bundle partitions the 27 RedCode-Exec categories into four groups based on declared mechanism and CWE class.The partition covers every execution category exactly once, and operators select a matching bundle before deployment.
Benchmark and baselines: Baselines include dual-agent runtime defenses, task-constraint checking, provenance marking, and Llama Guard 3 input classification.These defenses operate at different stages of the agent pipeline.

C Preamble Ablation

The preamble contributes little to measured safety outcomes, while the synthesized skill body accounts for nearly all observed effects. Under jailbreaks, concatenation sometimes improves bundle performance, but not consistently across models.

Preamble ablation: The preamble is the short refusal directive introduced in Section 5.1.Table 9 evaluates its removal using ASR, refusal rate, generation AvgS, and VirusTotal detection rate.
Preamble ablation: At most 1.7 percentage points of execution ASR and 0.013 generation AvgS change when the preamble is removed.Every refusal-rate change remains below 2.5 percentage points.
Preamble ablation: The synthesized body accounts for nearly all measured safety effects, while the preamble supplies a consistent refusal directive at negligible measured cost.Removing it lowers Reactive execution ASR from 3.8% to 2.2%, while generation safety remains nearly unchanged.
Concatenation control: Concatenation performs better on three of six models under each jailbreak family, while Seed is the only model never improved by concatenation.The comparison is between synthesized bundle skills and budget-matched concatenations.
Synthesis prompts: Both synthesis strategies share the refusal explanation, four writing guidelines, and a fixed YAML-plus-Markdown output structure.They differ in role description, input data, and one strategy-specific guideline.
Synthesis prompts: Proactive skills emphasize technical checks such as patterns, paths, and command structures, whereas reactive skills target harmful intent across presentation formats.Reactive prompts are based on recorded failures and ask why malicious requests passed the agent’s safety judgment.
Refusal policy: The intended safety behavior is binary: refuse harmful requests completely without execution, and execute legitimate requests normally.Partial modification or execution can preserve harmful capability and create reusable scaffolding.

G Coverage Analysis of the Granularity Gap

The coverage analysis explains the granularity gap as a fixed-budget consequence: one document must distribute detection rules across many categories, whereas per-class skills receive a full budget for the known category. The analysis also identifies empirical and statistical boundaries on the evaluation.

Empirical gap: 52.7% all-classes ASR versus 3.8% per-class ASR on DS-V3.2 yields a 48.9-percentage-point granularity gap.The paper interprets this difference through a fixed-character-budget coverage bound.
Coverage model: A skill of length L encodes at most L/ℓ̄ detection rules, with semantic generalization coefficient c limiting the signatures those rules jointly match.All-classes shares one budget across all K = 27 categories, while per-class gives each category a separate budget.
Coverage model: The per-class bound assumes the environment’s threat category is known before deployment and the corresponding skill is loaded for every request.D denotes the fraction of signatures not recognized by the active skill.
Theoretical bound: Under equal-size categories and optimal encoding, the largest bound difference is 1 − 1/K, which approaches 1 as the number of threat categories increases.The proof applies the rule-capacity limit separately to all-classes and per-class configurations.
Calibration: The DS-V3.2 calibration estimates approximately 100 rules per 10,000-character skill and about 20 signatures per category.Using the observed 52.7% all-classes ASR gives c ≈ 2.55, but the paper describes this as an algebraic consistency check.
Scaling predictions: At fixed budget, the analysis predicts that reducing category count lowers the all-classes ASR floor approximately as 1/K, while increasing body length lowers all-classes ASR.The predictions remain untested and per-class ASR should stabilize after signature coverage saturates.
Statistical evidence: On execution, 54 of 60 corrected clean-prompt comparisons reach p < 0.01, while generation reaches that threshold in 31 of 60 comparisons.All significant comparisons have large effects; generation has limited test resolution across eight malware families.
Comparative results: Reactive per-class reduces execution ASR by 32.6 percentage points versus Llama Guard, with a 95% bootstrap interval of [27.7,37.6].Proactive per-class reduces it by 28.2 percentage points, with interval [23.3,33.2].