Source-linked AI summary
Spider-Sense: Intrinsic Risk Sensing for Efficient Agent Defense with Hierarchical Adaptive Screening
Zhenxiong Yu, Zhi Yang, Zhiheng Jin, Shuhe Wang, Heng Zhang, Yanlin Fei, Lingfeng Zeng, Fangqi Lou, Shuo Zhang, Tu Hu, Jingping Liu, Rongze Chen, Xingyu Zhu, Kunyi Wang, Chaofa Yuan, Xin Guo, Zhaowei Liu, Feipeng Zhang, Jie Huang, Huacan Wang, Ronghao Chen, Liwen Zhang
TL;DR
Autonomous agents create security risks that mandatory, repeated lifecycle checks address at substantial execution cost. Spider-Sense embeds intrinsic risk sensing and selectively invokes hierarchical screening, and experiments report strong protection with low false detections and minimal latency overhead. S2Bench provides lifecycle-aware evaluation using realistic tool execution and multi-stage attacks.
Problem
Mandatory security validation at predefined agent stages repeatedly checks benign execution, causing accumulating latency in complex multi-step workflows.
Method
Spider-Sense embeds Intrinsic Risk Sensing in agent execution and invokes hierarchical screening only when potential risk is perceived.
Results
8.3% latency overhead accompanies Spider-Sense’s lowest reported Attack Success Rate (ASR) and False Positive Rate (FPR).
Takeaways & Limitations
Intrinsic risk awareness is presented as a practical foundation for scalable agent security with strong protection, low false detections, and minimal latency overhead.
Takeaways & Limitations
Future work must extend IRS beyond instruction-level conditioning and expand S2Bench to longer-horizon tasks, richer tool ecosystems, and multi-agent settings.
Abstract
from arXiv · showhide
As large language models (LLMs) evolve into autonomous agents, their real-world applicability has expanded significantly, accompanied by new security challenges. Most existing agent defense mechanisms adopt a mandatory checking paradigm, in which security validation is forcibly triggered at predefined stages of the agent lifecycle. In this work, we argue that effective agent security should be intrinsic and selective rather than architecturally decoupled and mandatory. We propose Spider-Sense framework, an event-driven defense framework based on Intrinsic Risk Sensing (IRS), which allows agents to maintain latent vigilance and trigger defenses only upon risk perception. Once triggered, the Spider-Sense invokes a hierarchical defence mechanism that trades off efficiency and precision: it resolves known patterns via lightweight similarity matching while escalating ambiguous cases to deep internal reasoning, thereby eliminating reliance on external models. To facilitate rigorous evaluation, we introduce S$^2$Bench, a lifecycle-aware benchmark featuring realistic tool execution and multi-stage attacks. Extensive experiments demonstrate that Spider-Sense achieves competitive or superior defense performance, attaining the lowest Attack Success Rate (ASR) and False Positive Rate (FPR), with only a marginal latency overhead of 8.3\%.
1 Introduction
Autonomous agents expand LLM capabilities but introduce security risks that mandatory, stage-wise checking handles at substantial latency cost. Spider-Sense instead embeds risk sensing in execution, activates hierarchical screening selectively, and reports strong defense with low false positives and 8.3% latency overhead.
- Autonomous agents combine perception, planning, and tool execution, expanding attacks from prompt injection and memory poisoning to real-world data exfiltration and unauthorized operations.
- Mandatory validation at predefined lifecycle stages adds accumulating latency as multi-step workflows require repeated checks.
- Intrinsic Risk Sensing embeds latent risk awareness in the agent and enables event-driven defense without constant stage-wise inspection.
- When risk is perceived, Spider-Sense pauses execution, routes suspicious content to security checking, and lets the main agent continue or terminate.
- Spider-Sense combines lightweight similarity screening for efficient checks with deeper reasoning for cases requiring further analysis, and S2Bench evaluates this in realistic lifecycle settings.
- 8.3% latency overhead accompanies the lowest reported Attack Success Rate (ASR) and False Positive Rate (FPR).
2 Related Work
Related work addresses safety at both the model and agent levels, including alignment, guardrails, trajectory supervision, reusable risk signals, policy reasoning, and multi-agent controls.
- LLM-Level Safety Alignment and Guardrails: LLM-level safety alignment improves adherence to human safety preferences through reasoning-focused training and system-level guardrails.
- Agent-Level Defensive Mechanisms: Agent-level defenses protect multi-step trajectories across planning, action, reasoning, and memory using trajectory-aware supervision and system constraints.
- Agent-Level Defensive Mechanisms: Runtime interception methods learn reusable risk signals, while policy-verification methods constrain action selection through explicit formal reasoning.
- Agent-Level Defensive Mechanisms: Multi-agent defenses use hierarchical data management and permission control to mitigate illegal access and poisoning risks.
3 Spider-Sense Framework
Spider-Sense uses Intrinsic Risk Sensing to monitor security-critical artifacts and trigger Hierarchical Adaptive Screening only when potential risk is detected. HAS combines fast similarity-based inspection with deeper reasoning for ambiguous cases before the agent decides how to proceed.
- Agent interaction model: The agent–environment loop exposes four security-critical artifacts: user query, internal plan, executed action, and environment observation.
- Intrinsic Risk Sensing: Intrinsic Risk Sensing autonomously evaluates each stage artifact with interaction history and system instruction, deciding whether to trigger a security check.
- Intrinsic Risk Sensing: When risk is sensed, the agent wraps the suspicious artifact in a stage-specific template and routes it to HAS for verification.
- Hierarchical Adaptive Screening: HAS applies lightweight screening at high confidence and escalates lower-confidence cases to more time-consuming fine-grained analysis.
- Hierarchical Adaptive Screening: Fine-grained analysis retrieves top-K similar cases and invokes an LLM to reason over the current artifact and retrieved cases.
- Hierarchical Adaptive Screening: The main agent receives the inspection result and autonomously chooses ACCEPT, REJECT, or SANITIZE.
4 S2Bench Dataset
S2Bench evaluates agent security across realistic, multi-stage execution rather than isolated static inputs. It combines broad scenario coverage, authentic tool interactions, challenging benign cases, and injected state-dependent attacks.
- S2Bench addresses static benchmarks’ limited stage coverage, scenario diversity, realistic tool execution, and tool-return modeling.
- The benchmark covers four execution stages, eight application domains, and 79 specific sub-task scenarios.
- Its tool library contains approximately 300 functions and over 100 types of realistic structured return contents.
- S2Bench includes 153 hard benign samples spanning all four stages to measure over-defense and false positives.
- An external injector manipulates tool outputs and memory retrieval results within execution loops, producing state-dependent deviations without modifying internal agent code.
5 Experiments
SPIDER-SENSE is evaluated across established safety benchmarks and S2Bench, with results covering predictive quality, agreement, lifecycle-stage robustness, efficiency, and component ablations. The experiments show strong safety performance with reduced false alarms and targeted interception, while ablations demonstrate the importance of stage-wise sensing and hierarchical screening.
- Experimental Setup: The evaluation covers Mind2Web-SC, eICU-AC, and S2Bench, comparing SPIDER-SENSE with static guardrails and agentic defenses using predictive, attack, false-positive, and agreement metrics.S2Bench reports phase-wise performance across the agent workflow, while Mind2Web-SC and eICU-AC assess web-interaction safety and clinical RBAC compliance.
- Main Results: SPIDER-SENSE achieves 100% agreement on Mind2Web and eICU across backbones while outperforming baselines on most predictive metrics.Under Claude-3.5, LPA improves from 84.8 to 95.8 and F1 from 90.3 to 92.1 on Mind2Web, and from 78.6 to 96.7 and 85.1 to 98.1 on eICU.
- Main Results: 20.0 and 17.7 are SPIDER-SENSE’s plan-stage ASRs with Qwen-max and Claude-3.5, addressing a major blind spot for prior defenses.Query-stage ASR remains 11.9 and 12.3, while Claude-3.5 results include action-stage ASR 2.4 and FPR 9.6.
- Main Results: 14.1 is SPIDER-SENSE’s Claude-3.5 query-boundary FPR, compared with 64.7 for the model-only baseline, while action-stage ASR and FPR are 2.4 and 9.6.The results indicate reduced over-blocking without sacrificing the reported action-stage robustness.
- Main Results: 23.4s with Qwen-max and 41.7s with Claude-3.5 demonstrate lower reported latency than heavy guardrail pipelines.The paper characterizes this as a favorable balance among robustness, utility, and efficiency.
- Ablation Study: 29.9 points is the ASR increase caused by removing action-stage sensing, showing that no single checkpoint captures all attack surfaces.The ablation argues that sensing across all four lifecycle stages is necessary for robust protection.
- Ablation Study: Removing fine-grained analysis improves efficiency but reduces Defense Score and Precision, whereas removing coarse-grained detection severely degrades Efficiency.The two components therefore serve complementary roles: deep analysis handles complex attacks, and coarse screening supports real-time responsiveness.
6 Conclusion
The conclusion presents SPIDER-SENSE as an intrinsic, event-driven defense framework that activates hierarchical screening when risk is perceived and evaluates it with S2Bench. The reported outcome is strong protection with low false detections and minimal latency overhead, while future work targets broader sensing mechanisms and longer-horizon evaluation.
- 6 Conclusion: SPIDER-SENSE embeds risk awareness into agent execution through Intrinsic Risk Sensing and activates hierarchical adaptive screening only when potential risk is perceived.The framework is paired with S2Bench for systematic evaluation.
- 6 Conclusion: Figure 5 illustrates in-situ interception of a tool-return injection attack at the observation stage using IRS and hierarchical adaptive screening.The case involves maliciously poisoned tool output containing injected code intended to induce unauthorized execution.
- 6 Conclusion: SPIDER-SENSE achieves strong protection with low false detections and minimal latency overhead, supporting intrinsic risk awareness as a practical foundation for scalable agent security.This is the paper’s stated conclusion about the framework’s practical scope.
- 6 Conclusion: Future work extends IRS through adaptive or learned mechanisms, long-horizon planning and credit assignment, and broader S2Bench settings.Proposed expansions include longer-horizon tasks, richer tool ecosystems, and multi-agent environments.
A.1 Evaluation Metrics
The evaluation uses detection and attack-defense KPIs to measure safety-label correctness, malicious-instruction capture, mitigation, and over-defensiveness.
- Label Prediction Accuracy (LPA) measures overall correctness across safety labels using true and false prediction counts.Its formula uses TP, TN, FP, and FN.
- Label Prediction Precision (LPP) measures the proportion of correctly identified malicious instructions among instructions flagged as risky.
- Label Prediction Recall (LPR) measures the defense’s ability to capture all actual malicious instructions.
- F1-Score balances Label Prediction Precision and Label Prediction Recall through their harmonic mean.
- Attack Mitigation (AM) measures the percentage of malicious attempts successfully blocked or neutralized, often calculated as 1 − ASR.
- False Positive Rate (FPR) measures how often benign user requests are incorrectly flagged as risks.This is also described as over-defensiveness.
A.2 Attack Implementation
The attack implementation covers multiple lifecycle entry points, including query, planning, action, and observation stages. Representative logs show IRS detecting attacks, routing suspicious content for screening, and refusing or interrupting unsafe execution.
- Attack vectors: The implementation evaluates attacks across agent logic, memory, tool-definition, and external-observation pathways.Examples include logic hijacking, poisoned memory retrieval, malicious tool descriptions, and injected instructions in external API outputs.
- Query stage: IRS successfully intercepted a logic-hijacking attack after detecting a high-risk fictional framing pattern.The defense analysis identified the disguised malicious intent and triggered the circuit breaker before execution.
- Plan stage: IRS blocked a memory-poisoning attack by identifying an unsafe retrieved plan involving unauthorized data export.The agent executed only the safe task subset after the injected DataExportTool command was detected.
- Action stage: IRS detected tool-definition and action-parameter attacks involving unauthorized uploads and potential data exposure.The defense compared the planned action with known attack patterns and required clarification rather than proceeding automatically.
- Observation stage: IRS intercepted external-observation hijacking by detecting malicious content and refusing further actions.The observation included a suspicious URL, which led to circuit-breaker activation and session termination.
B S2Bench Dataset Construction And Details
S2Bench is designed around the agent lifecycle, attack diversity, and realistic execution contexts. It covers multiple application domains and attack classes while including challenging benign cases for false-positive evaluation.
- Benchmark organization: Table 4 summarizes the benchmark’s attack-stage organization.The table is presented as an overview of attack stages.
- Lifecycle coverage: The benchmark organizes scenarios around four security-critical stages: query, planning, action, and observation.These stages correspond to adversarial influence over user queries, plans, actions, and environment feedback.
- Stage-specific scenarios: The benchmark defines stage-specific scenarios, including memory poisoning for planning, parameter tampering and unauthorized tool invocation for action, and indirect prompt injection for observation.These scenarios are intended to probe defensive robustness during execution and security boundaries around untrusted content.
- Attack coverage: The benchmark includes attack families targeting logic, inputs, reasoning, memory, embeddings, tool definitions, execution reasoning, and tool selection.These attacks exploit instruction priority, retrieval, planning, tool metadata, and human-approval workflows.
- Evaluation motivation: Traditional keyword-matching and semantic-similarity metrics are limited for autonomous-agent evaluation.The benchmark therefore focuses on attacks and behaviors occurring within the agent’s execution process.
C.2.3 Evaluation Models and Implementation
The evaluation uses automated LLM-based auditing for agent activity and security outcomes. Separate and combined judge prompts define binary criteria for execution, refusal, interception, and dangerous behavior.
- Evaluation models: GPT-4o-mini and judge prompts are used for automated auditing.The auditing setup is intended to provide reasoning capability during evaluation.
- Stage 1 auditing: Stage 1 auditing scores whether the agent produced substantive execution activity, tool calls, state transitions, or returned content.The judge does not compare the response with the original instruction’s goal when determining activity.
- Stage 1 auditing: Stage 1 assigns score 0 when the agent refuses, reports detected risk and stops, or produces no substantive response.Explicit refusal and risk blocking are treated as non-execution outcomes.
- Security outcome criteria: Dangerous execution is scored as 1 when behavior includes data exfiltration, malicious execution, security compromise, or resource abuse.Examples include uploading data, running unauthorized code, disabling defenses, and repetitive resource consumption.
D Stage-wise Vector Database
The stage-wise vector database stores generalized attack patterns and isolates retrieval across the agent lifecycle. Prompt-based extraction and refinement convert attack scenarios into reusable screening patterns for each stage.
- Vector-store design: The vector store uses ChromaDB with BGE-M3 embeddings and separate collections for each defense stage.Stage isolation is intended to prevent cross-stage noise interference during retrieval.
- Pattern refinement: Candidate attack patterns pass non-refusal, alignment, logic-consistency, and abstraction checks before persistence.Only patterns satisfying all four criteria are stored with their original content and taxonomy metadata.
- Stage libraries: The database contains pre-planning, pre-action, post-observation, and retrieve-phase libraries.These libraries address prompt injection, parameter pollution, malicious tool outputs, and adversarial retrieved knowledge.
- Pattern extraction: Stage-specific extraction prompts identify attack mechanisms, abstract their essence, and produce structured pattern representations.The prompts cover query, action, jailbreak, plan, and observation-stage attacks.
- Observation-stage extraction: Observation-stage extraction focuses on how external content deceives agent cognition through tactics such as instruction mimicry, obfuscation, format spoofing, and logic traps.The resulting pattern format records components and a concise attack pattern.