PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents

Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, Yongfeng Zhang

arXiv:2410.02644v4cs.CRcs.AI

TL;DR

LLM agents gain capabilities from external tools and memory, but their security is not comprehensively evaluated across operational stages. ASB formalizes and benchmarks attacks and defenses across diverse scenarios, agents, tools, and LLM backbones. It finds substantial vulnerabilities and limited defense effectiveness, motivating stronger agent security research.

  • Problem

    Existing research emphasizes agent effectiveness and generalization, while comprehensive evaluation of attacks and defenses across agent security stages remains limited.

  • Method

    ASB formalizes and evaluates diverse attacks and defenses across 10 scenarios, 10 agents, over 400 tools, 27 attack or defense types, and 13 LLM backbones.

  • Results

    ASB reveals vulnerabilities across system prompts, user prompt handling, tool usage, and memory retrieval, with the highest average attack success rate exceeding 84.30%.

  • Takeaways & Limitations

    ASB provides a resource for developing stronger defenses and more resilient LLM agents.

  • Takeaways & Limitations

    The threat model assumes attackers can insert system prompts and attack tools but lack knowledge of the backbone LLM’s architecture, training data, and parameters.

Abstract

from arXiv · show

Although LLM-based agents, powered by Large Language Models (LLMs), can use external tools and memory mechanisms to solve complex real-world tasks, they may also introduce critical security vulnerabilities. However, the existing literature does not comprehensively evaluate attacks and defenses against LLM-based agents. To address this, we introduce Agent Security Bench (ASB), a comprehensive framework designed to formalize, benchmark, and evaluate the attacks and defenses of LLM-based agents, including 10 scenarios (e.g., e-commerce, autonomous driving, finance), 10 agents targeting the scenarios, over 400 tools, 27 different types of attack/defense methods, and 7 evaluation metrics. Based on ASB, we benchmark 10 prompt injection attacks, a memory poisoning attack, a novel Plan-of-Thought backdoor attack, 4 mixed attacks, and 11 corresponding defenses across 13 LLM backbones. Our benchmark results reveal critical vulnerabilities in different stages of agent operation, including system prompt, user prompt handling, tool usage, and memory retrieval, with the highest average attack success rate of 84.30\%, but limited effectiveness shown in current defenses, unveiling important works to be done in terms of agent security for the community. We also introduce a new metric to evaluate the agents' capability to balance utility and security. Our code can be found at https://github.com/agiresearch/ASB.

1 INTRODUCTION

LLM agents expand task capabilities through tools and memory but create multiple security access points. ASB addresses limited prior coverage with a broad benchmark of attacks and defenses across agent stages and scenarios.

  • Motivation: Existing benchmarks often assess only one attack type or a small number of scenarios.The cited examples include Indirect Prompt Injection and settings such as financial harm and data security.
  • Attack coverage: ASB targets system prompts, user prompt handling, tool usage, and memory retrieval with multiple attack and defense types.Covered attacks include DPI, IPI, Memory Poisoning, PoT Backdoor, and Mixed Attacks.
  • Benchmark scope: ASB evaluates security across 10 scenarios, 10 agents, over 400 tools, and 400 tasks.Tasks include aggressive and non-aggressive types; aggressive tasks measure refusal of risky instructions.
  • Evaluation: 27 attack and defense types are benchmarked across 13 LLM backbones using 7 metrics.The benchmark reports vulnerabilities and limited effectiveness of existing defenses.
  • Contributions: The PoT Backdoor Attack embeds hidden instructions in the system prompt and exploits the agent’s planning process.The attack is presented as a novel contribution of ASB.

2 RELATED WORK

Related work covers prompt injections, memory poisoning, and backdoors in LLMs and agents. These lines of work motivate broader security evaluation for agents across attack surfaces and operational settings.

  • Prompt injections: Prompt injection can directly target user prompts or indirectly manipulate an agent through its external environment.The related work distinguishes direct and indirect influence on agent behavior.
  • Memory poisoning: Memory poisoning injects malicious or misleading data into memory or RAG databases for later retrieval and processing.Prior studies cited here mainly examine poisoning effects on LLMs and RAG.
  • Backdoors: Backdoor attacks embed triggers that produce harmful outputs, with prior work disrupting reasoning or contextual learning.The passage cites research targeting both LLMs and, more recently, LLM agents.

3 DEFINITIONS TO BASIC CONCEPTS AND THREAT MODEL

The threat model defines how agents retrieve memory, generate plans, and execute tool actions, then specifies attacker goals and capabilities. It distinguishes benign behavior from malicious redirection under constrained attacker knowledge.

  • Agent and memory: RAG agents retrieve the K most relevant query-solution memories from a database using shared query-key embeddings.Retrieved memories serve as in-context examples for the LLM.
  • Agent and memory: The backbone LLM uses the system prompt, query, observations, tool list, and retrieved memories to generate an action plan.The agent follows the generated plan through tool-using actions during execution.
  • Basic concepts: A target task consists of an instruction, tool list, and data supplied to an LLM agent in a specified format.The task is denoted by its target instruction, tools, and target data.
  • Basic concepts: Injected tasks redirect the agent from the target task toward an attacker-selected instruction, tool list, and data.Both direct and indirect prompt injections pursue this redirection.
  • Threat model: The adversary seeks to maximize the probability of a malicious action while PoT backdoors should preserve benign actions on clean queries.Under benign conditions, the agent is expected to perform the labeled benign action.
  • Threat model: Attackers know attack-tool details and can add malicious tools, but lack knowledge of the backbone model’s internals.They can also craft and insert prompts into the agent’s system prompt.

4 FORMALIZING ATTACKS AND DEFENSES IN LLM AGENTS

Section 4 formalizes attacks and defenses across the system prompt, user prompts, memory retrieval, and tool usage of LLM-based agents. It defines direct and indirect prompt injections, memory poisoning, and Plan-of-Thought backdoors, alongside mixed attacks and corresponding defenses.

  • 4 FORMALIZING ATTACKS AND DEFENSES IN LLM AGENTS: LLM agents expose system prompts, user prompts, memory retrieval, and tool usage as distinct attack surfaces.The framework also considers combinations of these attacks as mixed attacks.
  • 4.1.1 DIRECT PROMPT INJECTION ATTACKS: Direct Prompt Injection appends an injected instruction to the target user prompt and adds attack tools so the agent performs the injected task.The malicious action is achieved when the agent successfully uses all attack tools in the injected tool list.
  • 4.1.2 INDIRECT PROMPT INJECTION ATTACKS: Indirect Prompt Injection embeds an injected instruction in an observation from the execution trajectory, such as a tool response, while adding attack tools.This shifts the injection point from the user prompt to the agent’s accessible external environment.
  • 4.1.3 ATTACK FRAMEWORK FOR DIFFERENT PROMPT INJECTION WAYS: The prompt-injection framework represents compromised data as ˜x = P(xt, xe), where P produces maliciously modified prompt or observation data.For DPI, xt is the target instruction prompt; for IPI, xt is an observation result oi from the execution process.
  • 4.2 MEMORY POISONING ATTACKS: Memory poisoning supplies a poisoned RAG database and attack tools so the agent performs an injected task in addition to its intended task.The poisoned database combines clean memory with adversarial key-value pairs whose values are poisoned plans retrieved by query and tool-list similarity.
  • 4.3.1 DETAILED ADVERSARIAL GOAL: The PoT backdoor attack injects backdoored planning demonstrations into the system prompt, a trigger into the query, and attack tools into the available tool list.Its demonstrations connect a trigger with a backdoored planning step and adversarial target action.
  • 4.3.2 ATTACK FRAMEWORK: PoT backdoor construction uses planning steps as links between user prompts and adversarial actions, while triggers are designed with minimal semantic relevance.The experiments use non-word tokens or phrase-based triggers, including examples such as ‘@ @’ and ‘:)’.
  • 4.4 DEFENSES: ASB compares attack and defense mechanisms across its benchmark statistics, while Table 2 organizes defenses by the attacks they address.The defense section distinguishes prevention-based methods from detection-based PPL and LLM-based detection.

5 EVALUATION RESULTS ON AGENT SECURITY BENCH (ASB)

ASB evaluates attacks and defenses across diverse agent scenarios, tools, operational stages, and LLM backbones. Results show substantial attack effectiveness, capability-dependent vulnerability, and limited defense success.

  • Attack results: 84.30% average ASR makes Mixed Attack the most impactful attack, while Memory Poisoning has the lowest average ASR at 7.92%.Mixed Attack also has a 3.22% refusal rate; these results aggregate attacks across LLM backbones.
  • Backbone analysis: ASR and agent utility rise and then fall with backbone capability as instruction-following vulnerability is partly offset by stronger refusal mechanisms.GPT-4o records 60.35% ASR and 20.05% refusal under DPI, whereas GPT-3.5 Turbo records 98.40% ASR and 3.00% refusal.
  • Backbone analysis: Net Resilient Performance identifies backbones that balance agent utility and security.The metric is used to select suitable candidates for efficient and resilient real-world agent applications.
  • Backbone analysis: Agent performance is generally weaker than the standalone leaderboard quality of its backbone LLM.Most models fall below the red y = x line comparing backbone leaderboard quality with average agent performance.
  • Defense results: Current prevention-based defenses remain inadequate: Paraphrasing leaves DPI ASR at 56.87%, while Dynamic Prompt Rewriting leaves it at 44.45%.The defenses can also cause utility losses on primary tasks without attacks.

6 CONCLUSION AND FUTURE WORK

The paper formalizes attacks against different operational stages of LLM agents and introduces ASB to evaluate them across agent security settings. Its attack taxonomy includes direct, indirect, memory-based, planning backdoor, and mixed attacks.

  • Conclusion and future work: ASB targets user prompting, tool use, and memory retrieval, while mixed attacks combine threats across operational steps.PoT backdoor prompts target the system prompt and are excluded from mixed attacks because they are not recorded in the database.
  • Agent framework: ReAct agents receive observations, execute actions, and use language-space thoughts to update context for further reasoning or action.The framework models the policy over the current interaction context and extends the action space with thought actions.
  • Attack taxonomy: Five prompt-injection forms are formalized: Naive, Escape Characters, Context Ignoring, Fake Completion, and Combined attacks.These attacks append or structure injected instructions to manipulate how the system interprets the original task.

A.3.2 ATTACKING EXAMPLES

The examples illustrate how direct and indirect prompt manipulation, backdoored planning, and poisoned memory can redirect agents toward malicious tools. The defenses shown rewrite or delimit inputs and shuffle planning demonstrations to disrupt these attack paths.

  • Direct Prompt Injection: Direct Prompt Injection adds a malicious command to the user prompt, causing the agent to incorporate Data Export into its plan and leak financial reports.
  • Indirect Prompt Injection: Indirect Prompt Injection alters a tool observation after a correct first action, redirecting the next step toward Data Export and financial-report leakage.
  • PoT Backdoor Attack: PoT Backdoor embeds hidden instructions in system-prompt demonstrations so a trigger phrase causes the agent to generate a corrupted plan and invoke an unintended attack tool.
  • Memory Poisoning Attack: Memory Poisoning stores a malicious plan that is retrieved for a similar task, inducing incorrect actions that invoke Data Export to leak financial reports.
  • Defenses: Dynamic rewriting and paraphrasing transform injected queries, while delimiters constrain processing to the user instruction and Shuffle rearranges PoT planning steps.

B.3 BENCHMARK COMPARISON

ASB broadens agent-security evaluation beyond the narrower attack and scenario coverage of InjecAgent and AgentDojo. It compares diverse attacks and defenses across multi-domain agents, tools, tasks, and LLM backbones using simulated calls for reproducibility.

  • Benchmark Scope: ASB evaluates 16 attack methods and 11 defenses, covering more attack and defense types than InjecAgent and AgentDojo.
  • Benchmark Scope: ASB includes memory poisoning, PoT backdoors, and mixed attacks, whereas the compared benchmarks focus primarily on prompt injection and omit broader threat types.
  • Evaluation Setup: ASB evaluates both open-source and closed-source LLMs, including LLaMA, Gemma, Mixtral, Qwen, GPT, and Claude models.
  • Evaluation Setup: Experiments use simulated tool calls rather than real-world APIs to reduce variability from updates, maintenance, and temporary outages.

C.2.3 EVALUATION METRICS

ASB measures attack success, clean task performance, refusal behavior, and the balance between utility and robustness. Its Net Resilient Performance metric combines clean performance with resistance to successful attacks.

  • Core Metrics: Attack Success Rate measures how often an agent successfully invokes the attacker-selected malicious tool across attack tasks.
  • Core Metrics: Performance under No Attack measures benign task completion when the agent correctly invokes all required labeled tools without attacks or defenses.
  • Net Resilient Performance: Net Resilient Performance is defined as NRP = PNA × (1 − ASR), combining clean effectiveness with attack resistance.
  • Benchmark Results: Across attacks, Mixed Attack has the highest average ASR at 84.30%, while Memory Poisoning has the lowest average ASR at 7.92%.

D.1.2 ANALYSIS OF DIFFERENT ATTACK COMBINATIONS.

Combined attacks generally outperform standalone methods, with DPI+MP offering high effectiveness at lower complexity than the full Mixed Attack. PoT backdoors remain effective across triggers while preserving closely matched benign and clean utility performance.

  • Attack Combinations: DPI+IPI and DPI+MP achieve better attack effectiveness than single attacks by combining vulnerabilities across prompt, context, and memory channels.
  • Attack Combinations: Mixed Attack reaches 84.03% ASR, while DPI+MP reaches 83.02%, making DPI+MP a near-optimal effectiveness–cost balance when minimizing overhead matters.
  • PoT Backdoors: PoT backdoor triggers, including symbols and phrases, produce exceptionally high ASRs that are nearly or exactly 100%.
  • PoT Backdoors: For GPT-4o, benign performance and performance under no attack remain closely matched in PoT-backdoored agents.
  • Attack Combinations: Combined Attack achieves the highest average ASR across the evaluated prompt-injection methods at 38.01%.

D.1.5 ANALYSIS FOR AGENTS PERFORMANCE IN (NON)-AGGRESSIVE SCENARIOS

Aggressive tasks show lower attack success rates than non-aggressive tasks, alongside higher refusal rates that may improve agent resilience.

  • 33.12% average ASR for aggressive tasks versus 38.98% for non-aggressive tasks indicates lower attack success in aggressive scenarios.The passage attributes this difference to greater refusal of aggressive instructions.
  • 8.31% average refusal rate for aggressive tasks versus 4.87% for non-aggressive tasks accompanies the lower aggressive-task ASR.Higher refusal rates for aggressive inputs likely help mitigate more attacks.

D.2.1 DEFENSES FOR DPI AND IPI

The evaluated defenses provide limited protection against DPI, IPI, and PoT backdoor attacks, while also slightly reducing benign agent performance. The analysis attributes failures to semantic processing that basic prompt transformations do not fully control.

  • D.2.1 DEFENSES FOR DPI AND IPI: Current defenses for DPI and IPI are ineffective, according to the benchmark analysis.The section examines delimiter, paraphrasing, instructional prevention, and sandwich defenses.
  • D.2.1 DEFENSES FOR DPI AND IPI: Delimiters fail to fully isolate malicious instructions because models may interpret inputs flexibly through context and semantics.The defense assumes strict boundary following, which many language models do not reliably exhibit.
  • D.2.1 DEFENSES FOR DPI AND IPI: Paraphrasing leaves attackers’ semantic intent intact, so the model may still execute the malicious instructions.Rewording disrupts linguistic form without changing the underlying meaning.
  • D.2.1 DEFENSES FOR DPI AND IPI: Instructional prevention and sandwich prevention can still permit undesired actions because later inputs or tool outputs retain influential malicious content.The sandwich defense cannot fundamentally break the connection between adversarial input and tool output.
  • D.2.1 DEFENSES FOR DPI AND IPI: 29.46% average PNA without defenses declines slightly when DPI and IPI defenses are applied in no-attack scenarios.The defenses therefore slightly hinder benign agent functionality.
  • D.2.2 DEFENSES FOR POT ATTACK: 42.12% to 29.06% average ASR reduction from Paraphrase defense against PoT backdoors remains insufficient to fully mitigate vulnerabilities.Shuffle and Paraphrase have minimal impact on benign performance; average PNA is 33.17% for Shuffle and 34.40% for Paraphrase.
  • D.2.2 DEFENSES FOR POT ATTACK: Paraphrase and Shuffle defenses remain limited against PoT backdoors because semantic trigger understanding and reasoning processes persist beyond surface form or step order.The proposed alternative is dynamic reasoning validation, which would flag steps misaligned with task objectives but adds computational complexity.

D.2.3 DEFENSES FOR MEMORY ATTACKS

Defenses against memory attacks detect many attacks poorly, and perplexity-based detection struggles to separate malicious from benign plans across thresholds. The analysis links these failures to limits of external-model reasoning and complexity-based assumptions.

  • D.2.3 DEFENSES FOR MEMORY ATTACKS: 66% of memory attacks are not detected on average, while 20% of non-malicious inputs are incorrectly flagged by LLM-based defenses.The average FNR is 0.660 and average FPR is 0.200, indicating high missed-attack rates despite relatively low false positives.
  • D.2.3 DEFENSES FOR MEMORY ATTACKS: External detection models may miss subtle changes in an agent’s reasoning chain or context, especially during complex multi-step tasks.These models may not fully understand the agent’s internal reasoning and task structure.
  • D.2.3 DEFENSES FOR MEMORY ATTACKS: PPL detection struggles to distinguish benign and malicious memory content because FNR and FPR remain relatively high across thresholds from 2.4 to 4.8.Lower thresholds produce excessive false positives, while higher thresholds do not provide a good FNR–FPR balance.
  • D.2.3 DEFENSES FOR MEMORY ATTACKS: Perplexity detection can fail because malicious commands may be simpler than normal multi-step tasks rather than having higher perplexity.The method assumes abnormal plans generally exhibit greater complexity than normal tasks.
  • D.2.3 DEFENSES FOR MEMORY ATTACKS: Contextual memory validation is proposed as an alternative that checks a plan’s logical consistency against broader task requirements.This approach aims to distinguish commands that deviate from expected patterns from coherent legitimate actions.

F REPRODUCIBILITY STATEMENT

The reproducibility package provides code, environment setup, configurable experiments, backend integrations, and predefined scripts for replicating ASB attacks and evaluations.

  • ASB source code includes scripts, configuration files, Docker setup, and implementations of multiple adversarial attacks.Supported attack scripts cover DPI, IPI, Memory Poisoning, and PoT Backdoor attacks.
  • Conda and Docker installation instructions support consistent environments on systems with or without GPU support.Dependencies are streamlined through requirements.txt.
  • YAML files define experimental configurations for LLM models and attack types, enabling tests across GPT-4, LLaMA, and other open-source models.Integrations include Ollama and HuggingFace.
  • ASB supports OpenAI, Claude, and HuggingFace backends with documented API-key and environment setup instructions.
  • Predefined attack scripts enable direct execution of adversarial experiments under different configurations.These scripts are provided to facilitate replication of the reported experiments.
Loading 2410.02644v4…