SkillAtlas: An Attack Trace Library for Agent Skills

1 Introduction

• Agent skills can be benign for one task but unsafe for another, with decisive evidence sometimes appearing in intermediate tool results, errors, or judge decisions.This context dependence challenges defenses built around stable artifacts or reproducible behavior.Source: §1 · p. 1
• Existing agent-safety benchmarks and skill-security studies largely report aggregate scores or final outcomes, leaving limited reusable evidence of risks unfolding across attempts and tool interactions.Execution context can shape safety failures.Source: §1 · p. 1
• SkillAtlas turns private agent-skill security reports into reviewed, public-safe, searchable trajectories that capture context and execution feedback for risk analysis and downstream defense reuse.The proposed uses include retrieval, regression testing, and guard supervision.Source: §1 · p. 1
• Figure 1 frames the shift from reusable evidence based on artifacts or single runs to reviewed case–trace–step records.The proposed representation centers evidence on execution trajectories.Source: Figure 1 · p. 2

2.1 Design Goals

• Trajectory-level evidence captures the context and execution feedback that explain how agent-skill risk emerges and evolves during interaction.SkillAtlas presents public trajectories as reusable defense memory for future risk analysis and defense development.Source: §2.1 · p. 2

2.2 System Workflow

• The trace-governance loop connects submission, parsing and normalization, redaction and review, and public release and reuse.The parser converts heterogeneous reports into case-, trace-, and step-level records; raw artifacts remain private while sanitized previews and raw bundles are reviewed locally.Source: §2.2 · p. 2
• Failed and technical rounds remain in the corpus because they can expose dependency errors, script paths, tool parameters, permission boundaries, and interface formats that shape later attempts.A case is one approved finding, a trace or round is one attack attempt, and a step is a prompt, assistant message, tool call, tool result, or judge event.Source: §2.2 · p. 2

2.3 Usage Cases

• Users can compare risks by category, skill, model, outcome, or tool interaction, then inspect traces and steps to understand a finding.The public interface includes corpus overview, case search, report submission, and detailed trajectory inspection.Source: §2.3 · p. 3, Figure 2 · p. 3
• Typical cases include data exfiltration through session handoff, path traversal after a failed call exposes interface details, and hardcoded-token misuse revealed by a polling result.The decisive evidence can occur before the final assistant answer.Source: §2.3 · p. 3

3.1 Risk Distribution

• 1,705 cases (56.6%) succeed; Bias/Manipulation, Poisoning, and Backdoors skew successful, while Data Destruction and DoS have larger failed/ignored shares.Data Exfiltration accounts for 1,548 cases (51.4%); Data Exfiltration and Bias/Manipulation together account for 75.8%. The corpus contains 3,014 cases, 6,589 trajectories, 233 skills, 18 execution models, and eight risk categories.Source: §3.1 · p. 3, Figure 3 · p. 4

3.2 Feedback Iteration

• 724 of 1,705 successful cases (42.5%) first succeed after a non-success first round, with later attempts often benefiting from execution feedback.The figure’s line shows median error steps before first success.Source: §3.2 · p. 4, Figure 4 · p. 4

3.3 Tool-Level Evidence

• 74.1% of cases contain shell/process evidence and 61.8% contain file evidence; coverage also varies by risk type.Data Exfiltration, Malware/Ransomware, and Data Destruction rely more on shell/process and file operations, while Phishing and Bias/Manipulation contain relatively more web, session, or memory evidence.Source: §3.3 · p. 4, Table 1 · p. 4

3.4 Downstream Guard Reuse

• 0.770 accuracy and 0.769 Macro-F1 (Llama-3.1-8B) exceed the best comparison baselines by 26.0 percentage points and 0.266, respectively.The models were trained with trajectory-grounded labels and GRPO.Source: §3.4 · pp. 4–5

4 Conclusion

• SkillAtlas transforms private security reports into reviewed, public-safe trajectories while preserving context and execution feedback for risk analysis and downstream defense reuse.The paper characterizes the public corpus as showing heterogeneous, feedback-driven, tool-mediated agent-skill risks.Source: §4 · p. 5

Limitations

• The parser currently targets SkillAttack exports, so other red-team frameworks require adapters.The evaluation does not include a controlled user study or a deployment showing guard-induced reduction in attack success.Source: Limitations · p. 5

Ethical Considerations

• Rule-based redaction may miss unusual secrets, and sanitized summaries may still reveal attack strategy.The system defaults to private uploads, local-only review, redacted previews, and public-safe case payloads that exclude raw bundles, full trajectories, local paths, and skill archives.Source: Ethical Considerations · p. 5

• Agent security shifts its boundary from a file or process to interactions among model policy, tool permission, task context, and execution feedback.Source: A Related Work · p. 7
• Traditional defenses matched static indicators, then replayed suspicious artifacts in sandboxes, and later organized adversarial knowledge into reusable threat intelligence.Source: A Related Work · p. 7
• A reviewed trajectory records how prompts, model decisions, tool calls, observations, and feedback interact over time.Source: A Related Work · p. 7

A.1 Traditional Security Detection

• Static analysis detects stable artifact features without execution, while dynamic systems execute suspicious objects in controlled environments and observe process, file, and network behavior.Static-analysis examples include semantic patterns, package features, strings, signatures, and code structure; obfuscation limits this view.Source: §A.1 · p. 7
• Both paradigms expect risk to remain visible in a relatively stable object or reproducible behavior within bounded execution.Dynamic behavioral profiles can support clustering and family analysis, but retain the reproducibility assumption.Source: §A.1 · p. 7
• SkillAtlas changes the evidence unit because agent-skill risk spans skill text, models, tools, tasks, and observations, which static scans or one sandbox run can miss.Source: §A.1 · p. 7

A.2 Agent Skill Security

• ReAct interleaves model reasoning with environment actions, and Toolformer teaches models when and how to invoke external tools.Source: §A.2 · p. 7
• Tool-using agent systems make tool permission and execution context part of the safety boundary, while existing evaluation systems often center on task definitions, final labels, or aggregate scores.Examples include AgentDojo, Agent Security Bench, ToolEmu, AgentHarm, and MCPEval.Source: §A.2 · p. 8
• Attack studies examine risks from prompt injection, jailbreaks, malicious or poisoned skills, and closed-loop prompt refinement with execution feedback.These works locate risk in skill text, tool metadata, execution paths, and iterative feedback.Source: §A.2 · p. 8
• Defense systems address complementary stages through conversational controls, unsafe-input/output classification, agent diagnosis, and evaluation platforms.The remaining gap is lifecycle-oriented conversion of risk processes into reviewed, public, searchable, reusable trajectory evidence.Source: §A.2 · p. 8

B Detailed Case Study

• The medium-severity Bias/Manipulation case spans 28 steps over three rounds, with outcomes progressing from ignored to ignored to success.Source: B Detailed Case Study · p. 9
• In the Bias/Manipulation case, boldface marks prompt refinement and evidence decisive to each judgment; the Audio-Cog table keeps paths and identifiers sanitized.Source: B Detailed Case Study · p. 9, Table 2 · p. 9
• Table 2 presents verbatim excerpts from a three-round Audio-Cog trajectory: the first two rounds end in model-level refusal, while the third reaches the text-to-speech tool.Source: Table 2 · p. 9