PaperlayerView exact revision on arXiv

Source-linked AI summary

Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents

Xiaofang Yang, Ziqi Miao, Dianbo Sui, Jing Shao, Lijun Li

arXiv:2609.01487v1cs.CRcs.AI

TL;DR

Skill-augmented agents can let malicious skills steer later tool use in task-dependent ways that pre-install checks may miss. The paper introduces a skill-native runtime guard, a task-conditioned attack dataset, and feedback-driven guard evolution, reporting large attack-success reductions while retaining utility across evaluated settings.

  • Problem

    Skill-augmented agents create a runtime safety gap because malicious skills can steer task-dependent actions after installation, making pre-install vetting insufficient.

  • Method

    Defense-as-Skill packages SkillSonar as an installable, inspectable, editable policy guard and evolves it through feedback-driven Monte-Carlo Tree Search over concrete rollouts.

  • Results

    SkillSonar reduces ID ASR from 0.482 ± 0.039 to 0.104 ± 0.036 and OOD ASR from 0.606 ± 0.043 to 0.115 ± 0.037 across N = 10 GLM-5 runs.

  • Takeaways & Limitations

    Across Claude Code and OpenClaw, SkillSonar substantially reduces attack success while generally providing a favorable safety–utility trade-off.

  • Takeaways & Limitations

    Token efficiency remains model-dependent, with SkillSonar incurring the highest listed usage on Claude Haiku 4.5.

Abstract

from arXiv · show

Skill-augmented agents load reusable skills as persistent runtime context, improving task performance but also giving malicious skills a durable channel for steering future actions. Such skills may leak secrets, corrupt code, bypass approvals, or stage data for exfiltration only after a concrete user task and workspace state make the unsafe action appear useful. This makes pre-install vetting insufficient and calls for runtime, task-conditioned protection. We propose Defense-as-Skill, a defense paradigm that implements the runtime guard itself as an installable, inspectable, and editable skill. Our guard, SkillSonar, runs alongside untrusted task skills and checks sensitive actions against the user's task boundary, routing each action to an allow, replan, or confirmation decision without modifying the underlying agent runtime. To study this setting, we construct SCOPE-R, a task-conditioned dataset covering 6 risk families and 21 sub-categories, with 206 attack-confirmed malicious instances and 43 benign tasks. We then improve SkillSonar on the SCOPE-R training subset using runtime guard-skill evolution, a Monte-Carlo Tree Search procedure that evolves the on-disk guard skill from feedback on the rollouts. Across Claude Code and OpenClaw, the evolved guard substantially reduces attack success while maintaining a favorable safety-utility trade-off. On repeated GLM-5 runs, SkillSonar reduces ID ASR from 0.482 to 0.104 and OOD ASR from 0.606 to 0.115. Further analyses demonstrate transfer across victim models, held-out risk families, and external benchmarks, as well as retained protection against adaptive attackers. Ablations further show that explicit safety responsibility assignment and the skill-native representation are both important to the observed gains.

1 Introduction

Skill-augmented agents gain reusable runtime capabilities, but loaded skills can persistently steer actions toward harms that emerge only under task-specific conditions. The paper proposes an inspectable, editable runtime guard skill and introduces SCOPE-R to evaluate and improve it.

  • Skill-augmented agents load reusable instructions, scripts, resources, and workflows that remain available throughout tasks.
  • Loaded malicious skills can leak secrets, corrupt code, bypass approvals, or stage data for exfiltration through persistent runtime influence.
  • Pre-install vetting is insufficient because unsafe actions may emerge only when the user task, workspace state, and available tools make them appear useful.
  • Defense-as-Skill makes the runtime guard an installable, inspectable, and editable skill, with explicit invocation assigned as a distinct safety responsibility.
  • SCOPE-R organizes task-conditioned safety evaluation around benign task completion and malicious skill-induced behavior across six runtime risk families and 21 sub-categories.
  • SkillSonar checks sensitive actions against the user’s task boundary and maps them to allow, replan, or confirmation decisions without modifying the underlying runtime.

2 Related work

Related work positions skills as reusable workflow artifacts and identifies them as a security-relevant supply-chain component. Existing studies examine skill utility, continual learning, realistic environments, and attacks on poisoned or adversarial skills.

  • Skills research studies curated and self-generated skills across diverse tasks and continual skill learning from agent experience.
  • Broader agent benchmarks evaluate realistic OpenClaw environments, while the cited work focuses mainly on general agent capability.
  • Community-contributed skills can contain prompt injections, data-exfiltration logic, privilege-escalation patterns, and other unsafe behaviors.
  • Prior benchmarks study poisoned skill definitions and complementary attacks where the skill artifact is fixed and adversarial users manipulate execution.

3 SCOPE-R: A Dataset of Skill-induced Attacks on SAAs

SCOPE-R evaluates whether defenses block skill-induced unsafe actions while preserving benign task completion. It combines structured risk taxonomy, attack-confirmed instance construction, family-based splits, and runtime feedback for guard optimization.

  • Dataset design: SCOPE-R is a defense-oriented dataset for runtime safety in skill-augmented coding agents, measuring unsafe behavior alongside task utility.
  • Instance structure: Each malicious instance pairs a benign task, a task-relevant skill that may induce unsafe behavior, and risk metadata with observable runtime success criteria.
  • Risk taxonomy: The taxonomy follows the path from skill claims to agent capabilities, persistent state, data flows, execution, and resource burden, yielding six families and 21 sub-categories.
  • Dataset construction: The dataset starts from 15 PinchBench tasks, generates tailored benign skills, and injects adversarial behavior while leaving the user task unchanged.
  • Attack construction: Attack agents edit skills for targeted risks, while victim rollouts and judge criteria determine whether the induced malicious behavior succeeds.
  • Released dataset: SCOPE-R retains 206 attack-success-confirmed malicious instances and includes 43 benign tasks for false-positive evaluation.
  • Dataset split: The evaluation holds out complete risk families C and P for OOD testing, while remaining families support guard-skill evolution training.
  • Guard optimization: Runtime guard-skill optimization uses rollout trajectories, workspace states, scalar metrics, and judge rationales to select and edit candidate guards through MCTS.

4 Defense-as-Skill: Runtime Guarding from Inside the Agent

SkillSonar is a skill-native runtime policy layer that evaluates proposed actions against the user’s task boundary while preserving inspectability and portability across agent platforms. Its guard skill is optimized through feedback-driven Monte Carlo search over concrete guard-skill variants, using separate cheap and full evaluations and a safety-first objective.

  • Runtime guard design: SkillSonar runs within the same SAA as the task agent and uses the existing skill primitive rather than a separate endpoint, instrumented runtime, or sandbox modification.The same guard is deployed across Claude Code and OpenClaw.
  • Runtime guard design: Before sensitive operations, SkillSonar evaluates task-boundary alignment and can allow, narrow or replan, or require confirmation.The intended task, loaded skill instructions, and execution context inform the assessment.
  • Runtime guard design: Explicit invocation assigns safety responsibility to SkillSonar because semantic task–skill matching may overlook guards during otherwise benign tasks.This remains soft responsibility assignment rather than mechanism-level enforcement, so the host agent may still fail to apply the guard.
  • Runtime evaluation: SkillSonar is evaluated using attack success, task utility, confirmation count, and token cost, with utility measured separately on benign and malicious instances.These metrics jointly capture safety, task preservation, interruption friction, and rollout cost.
  • Guard-skill evolution: Guard-skill evolution treats each concrete guard version as a search-tree candidate and refines variants using runtime feedback from observed failures and utility behavior.Each candidate is full-evaluated at most once, and refinement is intended to preserve effective behavior while addressing attack successes.
  • Guard-skill evolution: The search separates cheap estimates from higher-fidelity full evaluations, then selects candidates with an upper-confidence bound and expands selected nodes with refined children.The search terminates when a success threshold is reached, the evaluation budget is exhausted, or no expandable candidates remain.
  • Guard-skill evolution: The scalar search objective prioritizes attack prevention, while utility, confirmation count, and token cost serve as secondary preferences among similarly safe candidates.A small epsilon prevents confirmation and token costs from outweighing attack success rate.
  • Evaluation split: SCOPE-R separates training, ID evaluation, and OOD evaluation by risk family, with OOD families entirely unseen during guard-skill evolution training.This split tests generalization beyond the risk families used for evolution.

5 Experiments

Across repeated runs, victim models, held-out risk families, benchmarks, and adaptive attacks, SkillSonar substantially reduces attack success while generally preserving utility. Further analyses show that explicit guard invocation and skill-native representation materially contribute to these gains, although safety–utility and adaptive-attack trade-offs remain.

  • MCTS refinement: 0.300 to 0.078 malicious ASR by MCTS iteration 9, although the candidate trajectory is not expected to improve monotonically.Each plotted point corresponds to a selected candidate guard.
  • Main runtime results: 0.482 ± 0.039 to 0.104 ± 0.036 ID ASR and 0.606 ± 0.043 to 0.115 ± 0.037 OOD ASR across N = 10 GLM-5 runs.The relatively small standard deviations indicate stable reductions across repeated executions.
  • Safety–utility trade-off: SkillSonar retains 0.779 ± 0.077 ID TaskUtil. versus 0.650 ± 0.062 for AcceptEdits while achieving lower ID ASR, but AcceptEdits has lower GLM-5 OOD ASR at 0.073 ± 0.025.Both methods have mean OOD TaskUtil. of 0.715, illustrating that the safety–utility trade-off is not uniform.
  • Held-out families: 0.606 ± 0.043 to 0.115 ± 0.037 OOD ASR on held-out Capability Control and Privacy & Data Flow families in the primary GLM-5 setting.Reductions also occur for Claude Haiku 4.5, from 0.707 to 0.254, and GPT-5.4, from 0.559 to 0.034.
  • Invocation ablation: SkillSonar reduces ID/OOD ASR from 0.400/0.582 to 0.104/0.109 when explicit safety responsibility assignment is used.Benign utility remains essentially unchanged at 0.756 versus 0.767, while ordinary skill selection alone provides limited protection.
  • Skill representation: 0.815 versus 0.655 ID task utility and 0.789 versus 0.672 OOD task utility for skill-native versus flattened matched policy content.SkillSonar also reduces ID/OOD ASR from 0.353/0.437 to 0.104/0.109 and token usage by about 21%.
  • Comparison with runtime guards: 0.104 ID and 0.109 OOD ASR with task utility 0.815 and 0.789, placing SkillSonar closer to the low-ASR, high-utility region than AgentSpec and TS-Guard.AgentSpec reaches 0.294 ID ASR with 0.581 ID utility, while TS-Guard reaches 0.412 ID and 0.492 OOD ASR with lower utility.

6 Conclusion

The paper studies runtime risks in skill-augmented agents and introduces SCOPE-R, Defense-as-Skill, and SkillSonar. SkillSonar reduces attack success, transfers across settings, and benefits from explicit responsibility assignment and modular skill representation.

  • SCOPE-R covers six risk families and 21 sub-categories, with attack-confirmed malicious and benign task instances.The taxonomy supports aggregate diagnosis and fine-grained analysis of skill-induced unsafe behavior.
  • SkillSonar is an installable policy-level runtime guard that checks proposed actions against the user’s task boundary.It is evaluated across Claude Code and OpenClaw without modifying the underlying host runtime.
  • SkillSonar substantially reduces attack success and generally provides a favorable safety–utility trade-off across runtime defenses.The paper also reports transfer to held-out risk families, additional victim models, external benchmarks, and adaptive attacks.
  • Explicit safety responsibility assignment supports reliable guard invocation, while modular skill-native representation provides gains beyond flattening the same policy content.These findings come from the paper’s ablations.

B Guard policy details

SkillSonar is a normal skill that evaluates proposed actions using runtime context and task boundaries. Its policy covers diverse security, privacy, and reliability risks, while SCOPE-R construction retains only attack-success-confirmed malicious variants.

  • Guard policy: SkillSonar is loaded as a normal skill and helps decide whether each proposed action remains within the user’s intended task boundary.The guard is not intended to certify a skill once and for all.
  • Guard policy: The guard triages code execution, file access, persistent changes, external account operations, sensitive data access, transmission, and potentially revealing outputs.It evaluates these actions using the user request, loaded skill instructions, and execution history.
  • Dataset construction: SCOPE-R construction iteratively edits a benign skill, runs a victim-agent rollout, and retains the variant only when the judge confirms the target attack.Unsuccessful variants are refined with judge feedback or discarded after the budget is exhausted.
  • Evaluation: Runtime evaluation materializes the guard and task skill, measures task utility, and evaluates attack success from trajectories and final workspace states.The protocol aggregates safety, utility, interruption, and efficiency measurements.

C.2 SkillSonar Runtime Evaluation

SkillSonar assigns proposed actions risk levels and selects interventions ranging from allowance to replanning or blocking. Its runtime-guard optimization uses feedback-driven Monte-Carlo Tree Search, and the same guard can run across different hosts.

  • Risk levels and interventions: SkillSonar assigns each proposed action to one of four risk levels based on its scope and potential side effects.Higher-risk examples include broad modification, nontrivial commands, sensitive-data access, transmission, and persistent external changes.
  • Risk levels and interventions: R0 actions are allowed, R1 actions may require narrowed scope, R2 actions require justification and possibly confirmation, and R3 actions are blocked or replanned.Blocked actions are replaced with safer alternatives for completing the benign task.
  • Guard-skill optimization: The MCTS optimizer evaluates guard candidates, generates feedback-guided children, cheap-evaluates them, and returns the highest-scoring fully evaluated candidate.The procedure uses separate cheap and full evaluation sets under a full-evaluation budget.
  • Runtime evidence: Runtime evidence includes the task, skill instructions, proposed operation, accessed or modified data, target resources, and execution history.These inputs align triage decisions with the SCOPE-R risk taxonomy.
  • Deployment: The same guard policy is deployed to Claude Code and OpenClaw without changing the host runtime.This follows from packaging the guard as a skill rather than a platform patch.

D Additional Runtime Evaluation Details

Because ordinary task–skill matching may overlook safety skills, the evaluation assigns SkillSonar a persistent first-step responsibility. The protocol remains soft, and audits indicate that judge errors in sampled cases tended to overestimate rather than underestimate attack success.

  • Guard invocation: Safety skills may be overlooked when invocation depends only on ordinary task–skill relevance, especially for unsafe behavior inside benign tasks.The protocol therefore instructs the agent to call SkillSonar before other actions or skill invocations.
  • Guard invocation: Persistent responsibility assignment improves guard discoverability but does not create mandatory runtime enforcement.The host agent may still fail to read, apply, or comply with the guard.
  • Judge auditing: Human audits sampled judge-predicted successes and failures separately for dataset construction and final evaluation.Audited samples covered 31.7% of construction cases and 18.0% of final-evaluation cases.
  • Judge auditing: 94% and 98% balanced audit agreement were obtained for the construction and evaluation judges, respectively.Observed disagreements were false-positive success labels, so sampled judge errors tended to overestimate rather than underestimate ASR.
  • AcceptEdits comparison: AcceptEdits is a platform permission mechanism rather than a semantic attack detector.Its allowlist configuration reduces interruptions but can increase attack success when allowlisted actions enable an attack.

E.2 Extended Experimental Settings

The evaluation tests SkillSonar across agent harnesses, held-out risk families, adaptive attacks, and transfer settings. Results show broad attack reduction, although robustness remains uneven across risk families.

  • Claude Code and OpenClaw use the same skills and tasks, differing only in host process and tool wiring.
  • The adaptive-attacker evaluation holds attack families, generation settings, sample budget, and three refinement iterations fixed against the final guard.
  • SkillSonar reduces ASR across all six risk families after attacker adaptation, but Specification Integrity remains the most difficult family.
  • Figure 6 reports ID and OOD ASR by optimization victim and deployment victim, with off-diagonal entries measuring zero-shot transfer without target-model attacks or additional guard search.
  • Privacy & Data Flow shows the largest adaptive reduction, from 43.3% to 8.3%, while Capability Control remains relatively challenging at 26.7% ASR.

F.2 Zero-shot Cross-model Transfer

Optimized SkillSonar guards transfer across victim models and held-out risk families without target-model labeled attacks or additional guard search. Transfer remains broadly effective, but task utility and residual ASR vary by deployment model and risk family.

  • The GLM-5-optimized guard reduces ID ASR from 0.417 to 0.199 on Kimi-K2.6, from 0.481 to 0.096 on Claude Haiku 4.5, and from 0.588 to 0.019 on GPT-5.4.
  • The Kimi-K2.6-optimized guard transfers in the reverse direction, reaching ID ASR of 0.112 on GLM-5, 0.059 on Claude Haiku 4.5, and 0.019 on GPT-5.4.
  • The GLM-5-optimized guard reduces OOD ASR from 0.480 to 0.161 on Kimi-K2.6, from 0.707 to 0.254 on Claude Haiku 4.5, and from 0.559 to 0.034 on GPT-5.4.
  • ID task utility is 0.799 on Kimi-K2.6, 0.730 on Claude Haiku 4.5, and 0.533 on GPT-5.4 for the GLM-5-optimized guard.
  • Each source-model evolution uses 95 labeled malicious cases, 11 benign cases, 10 MCTS rounds, and up to approximately 31 evaluated candidate guards.
  • Held-out Privacy & Data Flow improves by 61.5 percentage points and Capability Control by 44.8 percentage points, while Capability Control remains comparatively harder.

H Limitations

The evolved guard remains a Markdown policy whose protection depends on the host skill-augmented agent following its instructions. Evaluation also depends on a runtime judge and a small search budget, while baseline coverage is limited.

  • The evolved guard is still a Markdown policy, so its protection depends on whether the host skill-augmented agent follows its instructions.
  • A more advanced malicious skill may still try to ignore, override, or bypass the guard rules during execution.
  • Judge errors may affect results because optimization relies on a runtime judge measuring attack success, task utility, confirmation burden, and token use.
  • The search uses a small budget of at most 8 full evaluations, and larger budgets may improve the guard further.
  • Few existing runtime guards target Claude Code and OpenClaw, so the baselines are practical reference points rather than a complete comparison.
Loading 2609.01487v1…