Skynet: Workflow-Level Anomaly Detection for Agentic AI via Semantic and Structural Modeling

1 Introduction

• A single flawed or malicious step can propagate through downstream agents and tools, producing abnormal execution trajectories.The two origins include intrinsic errors and adversarial manipulation.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Existing defenses target specific failure classes or individual prompts and actions, leaving global workflow dependencies only indirectly examined.This leaves cross-step anomalies beyond any single specification or known pattern.Source: §1 · p. 2
• Workflow-level detection can capture non-local signals such as role-inconsistent delegation, atypical tool pathways, and abnormal artifact influence.The section argues for modeling both workflow semantics and dependency structure without assuming a particular attack or failure mode.Source: §1 · p. 2
• Skynet converts observed executions into directed workflow graphs and learns benign-only semantic and structural representations to detect deviations.It consumes structured runtime records and applies one off-manifold decision rule to attacks and failures, including previously unseen ones.Source: §1 · p. 2
• Evaluation uses three public benchmarks covering adversarial manipulation, intrinsic execution failures, and broad safety-in-the-wild.Training uses only benign trajectories from ETO and does not observe adversarial or failing workflows.Source: §1 · p. 2

2.1 System Model

• Modern agentic systems expose long-horizon workflows in which planners delegate subtasks, invoke tools, and exchange intermediate results.Skynet passively consumes existing runtime records rather than modifying agent logic, tools, protocols, or model weights.Source: §2.1 · pp. 2–3
• Workflow graphs represent observed steps as attributed nodes and encode temporal precedence, delegation, data flow, and cross-agent communication as typed edges.A protocol-agnostic schema supports heterogeneous runtimes through record parsing and adapters.Source: §2.1 · p. 3

2.2 Threat Model

• The threat model covers adversarial workflow manipulation and intrinsic execution failures, assuming trustworthy measurements and excluding attacks that forge identities or tamper with the detector.For external agents or services, the system assumes remote attestation; it also assumes a representative benign training corpus, which need not be exhaustive.Source: §2.2 · p. 3, §2.1 · p. 3

2.3 Problem Definition

• The problem is to learn a graph-level anomaly score and threshold from benign workflows alone, flagging whole workflows above a calibrated false-positive budget.The score must capture non-local regularities spanning multiple steps and agents rather than judging individual messages or actions.Source: §2.3 · p. 3

3.1 Overview and Design Rationale

• The framework is benign-only and graph-level by design, composing workflow construction, representation learning, and thresholded scoring into one class-agnostic pipeline.Its design rationale separates semantic context from dependency organization because anomalies may perturb either channel.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Workflow-Aware View Construction

• Invariant-aware views preserve benign variability in positive examples and create plausible counterfactual negatives by minimally violating workflow invariants.The negatives target role, provenance, dependency, or stability; replacing them with random graph augmentations reduces recall on near-manifold attacks and failures.Source: §3.2 · p. 5

3.3 Dual-Branch Workflow Representation Learning

• Separate semantic and structural branches encode execution attributes and dependency descriptors, then contrast invariant-preserving workflows against invariant-violating views.The branches decouple the two anomaly channels; training draws on in-batch benign negatives and same-anchor counterfactual negatives.Source: §3.3 · pp. 5–6

3.4 Workflow-Mode Anomaly Scoring

• The detector scores a workflow by its squared distance to the nearest of K benign execution modes and flags it when the score exceeds a benign-calibrated threshold.The multiple modes represent recurrent patterns such as coordination styles, tool-use regimes, or task-decomposition structures; the same rule covers unseen anomalies.Source: §3.1 · p. 4, §3.4 · p. 6

4.1 Evaluation Setup

• Evaluation covers Agent-SafetyBench, AgentErrorBench, and ATBench, plus a Combined scenario aggregating 808 anomalies with benign test workflows.The benign split is 4,494/562/562 train/validation/test; all detectors use the same graph inputs and benign training split, with baseline-specific threshold tuning.Source: §4.1 · pp. 6–7

4.2 Detection Effectiveness (RQ1)

• Benign workflows form a coherent manifold around 150 mode anchors, while anomalies from each benchmark lie off-manifold in the learned joint embedding.The UMAP view is illustrative; detection scores are computed in the original joint embedding.Source: §4.2 · p. 7, Figure 3 · p. 7
• Skynet achieves full recall across all 17 AgentErrorBench intrinsic failure types, while recall falls on ATBench cases most similar to benign workflows.On ATBench, recall remains high for overt attacks but drops for inherent agent failures and indirect prompt injection, motivating complementary content-aware checks.Source: §4.2 · p. 7

4.3 Workflow-Aware Learning Scheme: Component Contributions (RQ2)

• On Combined, removing structural encoding, counterfactual views, multi-mode scoring, or workflow-aware views worsens detection or calibration; single-mode scoring is most damaging.Removing counterfactual views raises FPR by roughly 1.5× beyond the 1% budget; collapsing 150 benign modes cuts recall by roughly one third, and removing workflow-aware views raises FPR by about 1.4× beyond budget.Source: §4.3 · pp. 7–8

4.2 Detection Effectiveness (RQ1)

• Skynet achieves 100% recall at 0.89% FPR on Agent-SafetyBench and AgentErrorBench, with the best recall–FPR trade-off and highest F1 on ATBench and Combined.Figure 4 shows Skynet dominating the deployment-relevant low-FPR region across all four scenarios; several baselines degrade on the broader scenarios.Source: §4.2 · p. 7, Figure 4 · p. 8, Table 1 · p. 8

4.4 Always-On Monitoring Overhead (RQ3)

• Online per-step latency ranges from 4.8–48.8 ms at steps 1 and 80 to roughly 0.1 s mean and 0.57 s P95 at intermediate steps.Offline per-workflow latency stays below one second even for the largest workflows; intermediate-step latency rises as graph reconstruction grows, then amortizes downward.Source: §4.4 · p. 8

• Existing safeguards isolate execution, enforce policies or verify actions, and monitor traces, but typically target specific violation types or rely on hand-crafted rules.Source: §5 · p. 8
• Graph anomaly methods provide generic graph signals, while Skynet adapts contrastive learning with workflow-aware invariants and counterfactual hard negatives, then uses multi-prototype inference.Source: §5 · p. 8
• LLM safety research has focused largely on prompt-level vulnerabilities and filtering or robust-training defenses [13] [45] [8] [16].Source: §5 · p. 8
• Skynet uses Agent-SafetyBench [48], AgentErrorBench [50], and ATBench [20] as external testbeds spanning adversarial manipulation, non-adversarial failures, and broad safety-in-the-wild settings.Source: §5 · p. 9

• Skynet represents execution traces as directed workflow graphs capturing semantic context and structural dependencies, then detects attacks, intrinsic failures, and zero-day anomalies using a benign-only multi-mode manifold.Source: §6 · p. 9
• Table 4 organizes runtime overhead by offline per-workflow cost, scalability with workflow size, and online incremental per-step latency.Source: Table 4 · p. 9
• High recall at a sub-1% false-positive rate, with per-workflow and per-step latencies suitable for online monitoring, summarizes Skynet’s reported benchmark outcome.Source: §6 · p. 9