PaperlayerView exact revision on arXiv

Source-linked AI summary

The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey

Juhee Kim, Xiaoyuan Liu, Zhun Wang, Shi Qiu, Bo Li, Wenbo Guo, Dawn Song

arXiv:2603.11088v1cs.CRcs.AI

TL;DR

Agentic AI systems combine LLMs with traditional software, creating security challenges that extend beyond component-level defenses. This paper systematically characterizes agent designs, taxonomizes attacks, surveys defenses, and uses case studies to show that practical, general-purpose defenses remain largely elusive.

  • Problem

    Agentic AI systems introduce security challenges from combining LLMs with traditional software, while existing component-level defenses do not address the full systems perspective.

  • Method

    The paper systematizes agent security through design dimensions, attack and risk taxonomies, defense mechanisms, and case studies informed by established security frameworks.

  • Results

    The survey finds that agentic AI security research has mapped the problem space, but practical and general-purpose defenses remain largely elusive.

  • Takeaways & Limitations

    The paper identifies realistic evaluation frameworks and broader defense research as critical directions for securing agentic AI systems.

  • Takeaways & Limitations

    Frequent validation prompts can cause decision fatigue, while current alert mechanisms often assume substantial user security literacy.

Abstract

from arXiv · show

AI agents that combine large language models with non-AI system components are rapidly emerging in real-world applications, offering unprecedented automation and flexibility. However, this unprecedented flexibility introduces complex security challenges fundamentally different from those in traditional software systems. This paper presents the first systematic and comprehensive survey of AI agent security, including an analysis of the design space, attack landscape, and defense mechanisms for secure AI agent systems. We further conduct case studies to point out existing gaps in securing agentic AI systems and identify open challenges in this emerging domain. Our work also introduces the first systematic framework for understanding the security risks and defense strategies of AI agents, serving as a foundation for building both secure agentic systems and advancing research in this critical area.

1 Introduction

Agentic AI systems combine LLMs with software components to enable autonomous tasks, but this flexibility creates security challenges that differ from traditional software and standalone models. The paper responds with a comprehensive systematization of agent design, attacks, defenses, and real-world gaps.

  • Motivation: Agentic AI integrates LLM reasoning with software components for applications including chatbots, software development, and web browsing automation.These systems support dynamic tool use and autonomous task execution.
  • Motivation: Recent incidents show prompt injection, remote code execution, data exfiltration, and web-agent attacks compromising repositories, systems, sensitive data, and banking accounts.These incidents demonstrate the severity and breadth of agentic AI vulnerabilities.
  • Research gap: Existing research largely targets individual attack vectors or components, leaving agent-wide security and interactions among multiple components insufficiently systematized.The paper identifies the need for a framework covering novel attack surfaces and system-level defenses.
  • Approach: The paper analyzes agent design dimensions, develops an attack taxonomy, surveys defenses, and uses real-world case studies including AutoGPT to identify gaps.Its approach examines security across the entire agent ecosystem rather than relying only on component-level defenses.
  • Contributions: The framework characterizes agents using seven design dimensions and organizes threats by adversary model and seven security-risk categories spanning the CIA triad.The dimensions are input trust, access sensitivity, workflow, action, memory, tool, and user interface.
  • Significance: The work presents a foundational, comprehensive system perspective intended to support researchers and developers building secure agentic AI systems.The authors describe it as the first systematic analysis of the agentic AI security landscape from this perspective.

2 Overview

The paper defines its scope around risks and defenses unique to, or amplified in, agentic systems and analyzes them at component and system levels. It combines agent design characterization, established taxonomies, defense-in-depth, and a systematic review of literature and web documents.

  • Scope: The survey focuses on security risks and defenses unique to, or significantly amplified in, agentic systems compared with traditional software and standalone LLMs.It emphasizes how autonomy and environment access magnify risks such as data exfiltration and unintended system manipulation.
  • Methodology: The methodology characterizes agents across seven design dimensions and analyzes agent-specific risks at both component and system levels.The framework distinguishes agent risks from non-agentic LLM risks such as jailbreaks and hallucinations.
  • Scope: Model-internal attacks such as model inversion are excluded because inference-time input-output operation does not fundamentally worsen them in agentic contexts.The scope instead prioritizes risks arising from agent structure, autonomy, and environmental access.
  • Methodology: The defense landscape applies defense in depth, informed by traditional system security, and draws on OWASP Top 10 for LLM Applications and MITRE ATLAS.The review covers academic literature and web documents on agentic AI security from 2023 to October 2025.

3 Design Landscape of Agentic AI Systems

AI agents are hybrid systems whose components and seven flexible design dimensions shape both capabilities and security exposure. Greater flexibility broadens functionality while increasing attack surfaces and risks.

  • 3.1 Design Components: AI agents combine LLMs with memory, tools, and an external environment to receive queries and execute action sequences.Retrieval tools collect external information, while execution tools modify the environment through actions such as writing files or sending emails.
  • 3.2 Design Dimensions and Security Implications: The seven design dimensions are input trust, access sensitivity, workflow, action, memory, tool, and user interface.Each dimension represents a continuous flexibility spectrum, although representative levels are used for simplicity.
  • 3.2 Design Dimensions and Security Implications: Input trust, workflow, and access sensitivity span increasingly flexible configurations that improve adaptability but expose agents to less trusted data, dynamic execution, or more sensitive resources.The passages describe flexibility changes for external data sources, action sequences, and system access.
  • 3.2 Design Dimensions and Security Implications: Broader tool access, persistent memory, and richer interfaces enhance functionality and personalization while introducing supply-chain, memory-poisoning, leakage, and interface attack risks.Tool flexibility can culminate in arbitrary tool selection; memory can span multiple sessions; interfaces can extend across web, terminal, and IDE environments.
  • 3.2 Design Dimensions and Security Implications: Overall, more flexible agent architectures broaden the attack surface and enable more diverse attack vectors.The paper frames flexibility and security as a general trade-off across the design dimensions.

4 Attack Landscape of Agentic AI Systems

The paper organizes agent attacks by threat model and maps them to a system-wide taxonomy of risks. It shows that flexible designs and interacting failures can amplify consequences from model errors into data leakage, corruption, or resource drain.

  • 4 Attack Landscape of Agentic AI Systems: The attack landscape is analyzed through attack vectors, a security-risk taxonomy, design-dimension mappings, and interactions among risks.The analysis covers both the attack landscape and methods for constructing attack paths and payloads.
  • 4 Attack Landscape of Agentic AI Systems: Attack vectors are classified under external, user-level, and internal threat models according to attackers’ access during execution.The classification distinguishes execution-time access from assumptions about how attacks were developed.
  • 4.2 Security Risks: The taxonomy identifies risks including heterogeneous untrusted interfaces and unconstrained or unsafe data flow across agent components.Unconstrained data flow allows information to move from untrusted inputs to outputs such as responses and tool calls.
  • 4.2 Security Risks: Greater flexibility in input trust, memory, and tools increases attack surfaces, while dynamic workflows amplify wrong instruction following, unsafe data flow, and hallucination risks.The paper links external data, persistent memory, third-party tools, and LLM-defined execution to distinct risk categories.
  • 4.3 System-level Analysis of Agent Risks: Access sensitivity, action capability, and user interface determine the severity of consequence risks such as leakage, corruption, and resource drain.Expanding action capabilities can transform information disclosure into environment corruption, while interfaces introduce additional attack channels.
  • 4.3 System-level Analysis of Agent Risks: Risk interactions can cascade from attacker-controlled inputs through model failures to data exfiltration, corruption, or excessive resource consumption.EchoLeak demonstrates a chain from heterogeneous untrusted interfaces and unconstrained data flow to sensitive-data exfiltration without user interaction.
  • 4.4 Attack Methods: Existing attacks often rely on human-built attack paths and payloads, while newer work explores fuzzing and trained attack models for automated generation.Examples include manually setting injection points and generating malicious instructions from role-playing, delimiter-confusion, or instruction-reset patterns.

5 Defense Landscape of Agentic AI Systems

The paper organizes agent defenses around CIA security goals plus contextual security, covering mechanisms and risks across the agent ecosystem. It emphasizes a comprehensive defense landscape rather than isolated component protections.

  • The defense landscape examines security goals and categorizes mechanisms by the risks they cover.Table 2 summarizes these defense categories and associated risks.
  • Security Goals: Agent security goals include confidentiality, integrity, and availability, supplemented by contextual security for appropriate information flows.Contextual security adapts contextual-integrity principles to agent tasks and access decisions.

5.2 Runtime Protection

Runtime protection enforces security during execution through guardrails, information-flow controls, monitoring, and human validation. These mechanisms improve coverage across inputs, outputs, data flows, trajectories, and user approvals but face accuracy, latency, privacy, and usability trade-offs.

  • Input Guardrails: Input guardrails validate user inputs, retrieved results, and memory data before malicious content reaches agent internals.Agent-specific checks include validating retrieved data and constraining URLs with allowlists.
  • Output Guardrails: Output guardrails inspect agent responses, tool invocations, and action sequences to catch harmful outputs and unsafe behavior.They range from rules and classifiers to hybrid policy systems, but incur false positives, false negatives, latency, and computational costs.
  • Information Flow Control and Taint Tracking: Information-flow control and taint tracking label and propagate data through execution to detect unsafe flows affecting integrity or confidentiality.Multi-execution and variable-based methods can impose substantial runtime overhead, while conservative labeling can make agents unusable.
  • Monitoring: Monitoring provides system-wide visibility across inputs, outputs, intermediate states, tools, and long-running multi-agent interactions.Finer-grained logs may improve detection but increase storage, computation, and privacy exposure; stochastic behavior also complicates classification.
  • Human-In-The-Loop Validation: Human-in-the-loop validation lets users approve agent actions and tool use, with scope, alert content, and recurrence determining the interaction design.Frequent prompts can cause decision fatigue, while unclear alerts may lead users to approve risky actions or reject benign ones.

5.3 Secure By Design

Secure-by-design defenses establish security properties architecturally, including privilege separation and formal verification. These approaches isolate authority and constrain agent behavior, but formal verification remains difficult for probabilistic LLM-based systems.

  • Secure-by-design mechanisms make agents intrinsically secure through architectural principles tailored to agent-specific structures.Their designs depend on how components, privileges, and execution responsibilities are organized.
  • Privilege Separation: Privilege separation isolates agent components or environments to apply least privilege and reduce system-wide risk.Vertical separation assigns different authority levels, while horizontal separation isolates equal-privilege components by access scope.
  • Formal Verification: Formal verification encodes user intent, plan constraints, or safety properties to check whether agent behavior satisfies specified requirements.Examples formalize GUI state transitions or tool orderings, but traditional methods face challenges with probabilistic agent behavior.

5.4 Identity and Access Management

Agentic identity and access management extends traditional controls to delegated, dynamic interactions with services, memory, tools, and environments. The survey covers identity, delegation, access policies, and credential handling while highlighting gaps in adaptive and standardized protection.

  • Overview: Agentic IAM requires agent-specific identities, delegation mechanisms, and dynamic access policies that adapt to runtime context.Unlike traditional systems, agents act on users’ behalf across external services.
  • Identity Management: Identity management supports authentication, authorization, delegation, auditability, and accountability across centralized or decentralized architectures.Identity scope may be user-level, agent-level, or task-level, while delegation may be direct, proxy-based, or temporary.
  • Access Control: Access control constrains agent access to memory, databases, tool APIs, file systems, and inter-agent resources through RBAC, ABAC, capabilities, and dynamic policies.Multi-agent controls must also prevent confused-deputy attacks caused by delegated capabilities.
  • Credential Management: Credential management protects API keys, access tokens, one-time passwords, and session tokens through storage, lifecycle, and provisioning practices.The survey identifies ad hoc secret handling and unencrypted environment-variable storage as ongoing concerns.

5.5 Component Hardening

Component hardening strengthens individual models and tools against their specific vulnerabilities, following the principle that system security is limited by its weakest component.

  • 5.5 Component Hardening: Component hardening strengthens individual agent components, including models and tools, against their specific vulnerabilities.The approach treats models and tools as separate hardening targets within the agent system.

5.6 Defense Design Principles

Secure agent defense requires complementary mechanisms grounded in traditional security principles. Least privilege limits permissions, while complete mediation requires verification and authorization for sensitive-resource access.

  • 5.6 Defense Design Principles: Effective agent security requires multiple complementary defense mechanisms rather than reliance on a single approach.The paper frames agents as hybrid systems that inherit traditional security concerns while amplifying the need for coordinated defenses.
  • 5.6 Defense Design Principles: Least privilege restricts agents to the minimum permissions and access rights necessary for specific tasks.Privilege separation isolates components and limits tool access, while identity management defines appropriate agent access scopes.
  • 5.6 Defense Design Principles: Complete mediation requires every access to sensitive resources to be verified and authorized.The principle is presented as a defense requirement for controlling agent access to sensitive resources.

6 Securing Real-World Agents

The case studies show that real-world agents combine defenses unevenly across coding and web settings. Coding agents constrain actions and use human approval, while web agents face broad untrusted-input exposure and fragile protections.

  • 6 Securing Real-World Agents: The analysis examines six open-source agents and distinguishes fully supported defenses from partial coverage or defenses requiring substantial manual effort.The evaluation focuses on system-level defenses and records which defense classes each agent enables.
  • 6.1 Coding agents: Coding agents face hallucinated code, model backdoors, and hidden malicious instructions in seemingly trusted inputs.Potentially hostile inputs include user content, repositories, and documentation.
  • 6.1 Coding agents: Coding agents prioritize constraining actions over sanitizing inputs through output guardrails, access control, and human approval for sensitive operations.They gate filesystem operations and shell commands, while implementations differ in how they define sensitive actions and enforce permissions.
  • 6.1 Coding agents: Monitoring is only partially supported in coding agents, which log actions for post-hoc review without detecting suspicious patterns or anomalies.Collected events include tool calls and file modifications.
  • 6.2 Web agents: Web agents process arbitrary external content while handling sensitive data and authenticated workflows, creating a broad untrusted-input surface.Their tasks can include navigation, form completion, cloud-file access, email, and purchases.
  • 6.2 Web agents: Web agents commonly use input and output guardrails, credential management, and monitoring, but these defenses remain incomplete and fragile.Regex-based OTP protection raises the bar for exfiltration but does not cover other credential types; the agents remain early-stage prototypes.
  • 6.2 Web agents: Web-agent security gaps include missing information-flow control, identity management, and human oversight for high-impact actions.The proposed mechanisms would track untrusted-content influence, authenticate across services, and oversee purchases or data sharing.
  • 6.2 Web agents: Existing web-agent defenses need stronger automated domain assessment, contextual filtering, structural protections, and monitoring.The paper proposes combining domain reputation, taint tracking, privilege separation, and model-level guardrails.

7 Detailed Case Study: AutoGPT

The AutoGPT case study examines vulnerabilities arising from broad retrieval and execution capabilities, including malicious external content that reaches system operations. Patches mitigate downstream effects but leave upstream injection and unsafe data-flow risks open.

  • 7.1 AutoGPT Overview: AutoGPT combines retrieval and execution tools to interact with the Internet, local files, and system-level interfaces.Its tools include web search, webpage browsing, file access, memory loading, and shell execution.
  • 7.2 Real-world Vulnerabilities in AutoGPT: The case study analyzes five representative CVE vulnerabilities and maps them to the paper’s risk taxonomy.The vulnerabilities include container-configuration overwrite and path traversal triggered through malicious external content.
  • 7.2 Real-world Vulnerabilities in AutoGPT: Malicious instructions in fetched web content can hijack AutoGPT into overwriting container configuration, leading after restart to container escape and host compromise.The attack uses execute_python_code to modify docker-compose.yml before the compromised container is executed.
  • 7.2 Real-world Vulnerabilities in AutoGPT: Path traversal enables writes outside the sandbox, allowing modified source files to execute persistently and potentially expose sensitive source or configuration data.Attackers can induce a traversal path such as ../../main.py through instructions embedded in external content.
  • 7.3 Defenses in AutoGPT: AutoGPT patches target downstream consequences such as access control and output sanitization, leaving indirect prompt injection and unsafe data flow unaddressed at their source.The paper recommends input guardrails and information-flow control to block injection and prevent tainted outputs from reaching execution tools.
  • 7.3 Defenses in AutoGPT: The defense analysis records each CVE’s exploited risks, deployed patch, mitigated and open risks, and missing defense categories.This structure supports comparison between patch coverage and residual risk.
  • 7.3 Defenses in AutoGPT: Rule-based escape-sequence sanitization partially addresses unsafe data flow but may be bypassed through novel encodings or lesser-known control codes.The paper recommends treating all data from external web sources as untrusted and sanitizing it at every output boundary.

8 Conclusion

The survey maps AI agent attacks, defenses, risks, and open challenges, finding that practical general-purpose defenses remain elusive. It identifies realistic evaluation, composable and adaptive defenses, and standardized identity and access control as critical directions.

  • Practical and general-purpose defenses for agentic AI security remain largely elusive despite progress in mapping the problem space.
  • Realistic evaluation frameworks are needed to bridge research and production.
  • Composable defenses should avoid emergent misalignment while supporting standardized agent identity and access control.
  • Adaptive defenses should balance security with usability.
  • The survey aims to guide secure-agent development and identify meaningful directions for future research.
Loading 2603.11088v1…