PaperlayerView exact revision on arXiv

Source-linked AI summary

SoK: The Attack Surface of Agentic AI - Tools and Autonomy

Ali Dehghantanha, Sajad Homayoun

arXiv:2603.22928v2cs.CR

TL;DR

Agentic AI combines LLMs with tools, retrieval, and autonomy, but its composition creates security risks across trust boundaries, data, execution, and coordination. This SoK systematizes those risks through a threat taxonomy, attacker models, evidence synthesis, metrics, and defense guidance. It concludes that layered controls and continuous measurement are needed, while multi-agent and lifecycle defenses remain comparatively immature.

  • Problem

    Agentic systems broaden LLM applications into tools, retrieval, autonomous planning, and multi-agent orchestration, but their resulting attack surface and trust-boundary risks require focused systematization.

  • Method

    The paper develops a threat-driven taxonomy, defines trust boundaries and attacker models, synthesizes 2023–2025 evidence, maps risks to OWASP and MITRE ATLAS, and proposes metrics and defenses.

  • Results

    The synthesis identifies indirect prompt injection, RAG poisoning, tool and API abuse, and multi-agent propagation as recurring risks, with layered defenses reducing unsafe actions and limiting impact.

  • Takeaways & Limitations

    Security requires defense in depth across data, inference, planning, acting, and deployment, together with continuous attacker-aware measurement and CI-integrated red-teaming.

  • Takeaways & Limitations

    Post-exploitation persistence, fine-grained retrieval perturbations, and reproducible tool-misuse testbeds remain underexplored, while multi-agent and lifecycle defenses have mostly grade C evidence.

Abstract

from arXiv · show

Recent AI systems combine large language models with tools, external knowledge via retrieval-augmented generation (RAG), and even autonomous multi-agent decision loops. This agentic AI paradigm greatly expands capabilities - but also vastly enlarges the attack surface. In this systematization, we map out the trust boundaries and security risks of agentic LLM-based systems. We develop a comprehensive taxonomy of attacks spanning prompt-level injections, knowledge-base poisoning, tool/plug-in exploits, and multi-agent emergent threats. Through a detailed literature review, we synthesize evidence from 2023-2025, including more than 20 peer-reviewed and archival studies, industry reports, and standards. We find that agentic systems introduce new vectors for indirect prompt injection, code execution exploits, RAG index poisoning, and cross-agent manipulation that go beyond traditional AI threats. We define attacker models and threat scenarios, and propose metrics (e.g., Unsafe Action Rate, Privilege Escalation Distance) to evaluate security posture. Our survey examines defenses such as input sanitization, retrieval filters, sandboxes, access control, and "AI guardrails," assessing their effectiveness and pointing out the areas where protection is still lacking. To assist practitioners, we outline defensive controls and provide a phased security checklist for deploying agentic AI (covering design-time hardening, runtime monitoring, and incident response). Finally, we outline open research challenges in secure autonomous AI (robust tool APIs, verifiable agent behavior, supply-chain safeguards) and discuss ethical and responsible disclosure practices. We systematize recent findings to help researchers and engineers understand and mitigate security risks in agentic AI.

1 Introduction

Agentic AI systems expand LLM applications with tools, retrieval, planning, and coordination, creating a qualitatively different attack surface across model, data, execution, and lifecycle boundaries. This SoK systematizes those risks through a threat-driven taxonomy, evidence synthesis, metrics, defenses, and deployment guidance.

  • Motivation and scope: Agentic systems combine LLMs with tools, APIs, retrieval, autonomous planning, and multi-agent loops, while static pure Q&A is out of scope.The scope matches copilot deployments pairing LLMs with tool APIs and enterprise RAG.
  • Emerging threats: Indirect prompt injection, targeted RAG poisoning, and weak tool bridges expose agents to malicious instructions, code execution, traversal, and SSRF.One cited study found 19 RCE flaws across 11 frameworks with real exploits.
  • Emerging threats: The resulting attack surface combines model-level failures with traditional software and supply-chain vulnerabilities.Agentic composition connects prompt manipulation to tools, external data, and execution infrastructure.
  • Contributions and approach: The paper contributes a literature-driven taxonomy, attacker and trust-boundary model, graded 2023–2025 evidence corpus, attacker-aware metrics, defenses, and a practitioner playbook.Metrics include UAR, PAR, PED, RRS, TTC, OORAR, and CES.
  • Position within Prior Work: Its scope emphasizes agentic attack surfaces rather than exhaustive general LLM safety or other security aspects of AI agents.The analysis maps threats to OWASP GenAI and MITRE ATLAS and focuses on causal paths tied to privileged effects.

2 Background & System Model

The reference system repeatedly plans, selects tools, executes actions, and updates state while incorporating retrieved or externally supplied content. Security depends on treating model outputs and untrusted content as crossing critical boundaries and enforcing isolation, validation, and least-trust assumptions.

  • Reference pipeline: An agent receives a goal, optionally plans steps, selects structured tools, maintains context or memory, and repeats actions until completion, budget exhaustion, or refusal.Tools include browsing, sandboxed code execution, file and database access, external APIs, and inter-agent messaging.
  • Reference pipeline: RAG embeds a query, retrieves nearby corpus chunks, appends them to model context, and uses the resulting context to ground answers.The retrieval-and-action loop can continue across multiple tool invocations.
  • Trust boundaries: LLM-generated outputs must be validated before execution because the model is treated as an untrusted code generator at the execution boundary.The execution boundary separates free-form generation from external tools and environments.
  • Trust boundaries: Web pages, uploaded documents, and emails entering model context are hostile inputs, unlike vetted access-controlled corpora.This boundary makes content provenance and defensive handling central to the system model.
  • Trusted Computing Base: The TCB comprises model weights and system prompt, tool-execution isolation and validation, secret storage and credential issuance, and proprietary RAG indexing infrastructure.User inputs, external content, and plugin outputs are non-TCB elements that require defensive handling.
  • Security assumptions: The threat model assumes guardrails, filters, and sandboxing may exist but can be bypassed, so defense-in-depth is mandatory under worst-case inputs.Operators control schemas, sandboxes, retrieval, and policy while assuming malicious instructions may be followed.

3 Threat Model

The threat model represents adversaries, assets, and attack propagation through agent pipelines, including external content, supply-chain components, insiders, tools, and retrieval infrastructure. It uses attack-surface taxonomies and causal graphs to connect attacker techniques and preconditions to unsafe effects and defense points.

  • Threat model: The model defines adversary capabilities, affected assets, security properties, and causal propagation through agentic systems.A directed threat graph represents how compromised elements lead to subsequent states or actions.
  • Adversary classes: External attackers can submit prompts or malicious links, including content fetched unknowingly by an agent’s user.This class may operate without being a user of the agent system.
  • Attack taxonomy and causal graph: The taxonomy organizes attack vectors by goals, paths, and pipeline stages, while the causal graph identifies vulnerable or critical links where defenses can interdict attacks.Table 1 summarizes attack vectors and links goals G1–G7 with paths P1–P5.
  • Adversary classes: Supply-chain attackers target plugins, packages, model checkpoints, or vector databases, while insiders can misuse configurations, prompts, secrets, or corpora.A trojaned plugin may execute unauthorized actions when triggered by selected prompts.
  • Assets and Security Properties: Assets include sensitive data, credentials, infrastructure, behavioral integrity, and CIA properties, alongside policy integrity, accountability, and auditability.Attacks may leak information, alter outputs or actions, consume resources, violate policy, or undermine trustworthy logs.
  • Threat assumptions: The model assumes baseline sandboxing, content filters, and non-root deployment while examining how attackers can bypass or abuse these defenses.These assumptions avoid treating completely naive systems as the entire threat-model baseline.

4 Taxonomy of Attacks on Agentic AI Systems

The taxonomy organizes agentic-AI attacks by goals, vectors, pipeline stages, and composable attack paths. It covers prompt, retrieval, tool, API, memory, multi-agent, and supply-chain surfaces.

  • Attack goals: The taxonomy distinguishes enablers—privilege escalation, persistence, supply-chain compromise, and sometimes integrity subversion—from outcomes such as leakage, resource abuse, and fraud.
  • Prompt-level attacks: Prompt injection can target instructions directly or arrive through retrieved web pages, emails, and documents that the model treats as prompt content.
  • RAG-level attacks: RAG attacks poison corpora or retrieval, exploit embedding weaknesses, and may bypass authorization by surfacing documents beyond user entitlements.
  • Tool and API attacks: Tool and API attacks exploit ambiguous schemas, unsafe arguments, code execution, filesystem access, shell commands, over-privileged tokens, and cross-service connectors.
  • State, orchestration, and supply chain: Memory, multi-agent, and supply-chain attacks poison persistent state, forge or replay inter-agent messages, compromise dependencies, or tamper with models and plugins.
  • Composed attack paths: Attack paths compose vectors across stages, including direct prompts causing tool misuse and cross-tool pivots that prepare conditions for later abuse.

5 Evidence Synthesis of Attacks and Defenses

The evidence synthesis combines a structured review of recent research, standards, and practitioner sources with graded coding of attack and defense evidence. It finds recurring risks in tool use, indirect injection, RAG, and multi-agent systems, while defenses remain incomplete and evidence maturity uneven.

  • Corpus construction: The review screened approximately 100 candidates, retained approximately 40 for full-text review, and coded evidence type, system setting, and key claims.
  • Evidence grading: Sources were graded A–D by publication quality, artifacts or proofs, and reproducibility-related evidence such as live exploits or quantitative measurements.
  • Corpus coverage: The final corpus covers peer-reviewed papers, preprints, standards, industry documents, and selected technical blogs and artifacts from 2023–2025.
  • Key findings: LLMSmith reports 19 RCE vulnerabilities across 11 agent frameworks, with successful exploitation of real applications and later CVE remediation in multiple stacks.
  • Key findings: Live demonstrations and systematic benchmarks show that models do not reliably separate retrieved content from instructions; boundary tokens and reminders remain insufficient outside white-box control.
  • Key findings: Retrieval introduces additional failure modes and poisoning can yield high-success targeted responses, while multi-agent attacks can propagate through malicious messages despite partial vaccination-based mitigation.
  • Defenses: Surface sanitization and guardrails are bypassable, whereas typed APIs, execution isolation, human gating, monitoring, and anomaly detection reduce risk without guaranteeing robustness.
  • Consensus and disagreement: Prompt injection is widely treated as the primary enabling vector and least privilege as essential, but alignment and policy-prompt effectiveness varies across setups.

6 Evaluation & Metrics for Agentic AI Security

The paper proposes attacker-aware metrics computed from structured execution traces to evaluate agentic AI security across unsafe actions, policy adherence, escalation, retrieval, containment, and exploitation costs. It recommends controlled, reproducible, continuously updated evaluations using red-team scenarios, standardized harnesses, and independent adjudication.

  • Metric framework: Attacker-aware metrics capture instruction-following failures, tool/API side effects, and retrieval risks from structured execution logs.The paper notes pitfalls including non-determinism, evaluator leakage, and state carry-over.
  • Core metrics: Unsafe Action Rate measures the fraction of scenarios that trigger a policy-violating action, with variants for tool misuse and leakage.Lower UAR is better.
  • Core metrics: Policy Adherence Rate measures the proportion of executed actions complying with policy, but generally differs from UAR outside single-action adversarial tests.PAR approximates 1 − UAR only under one-action-per-scenario testing with crisp labels.
  • Path and response metrics: Privilege-Escalation Distance measures the shortest directed path from an untrusted input to a privileged action, with higher values indicating more barriers.The threat graph is static per deployment, with unit-cost edges unless otherwise specified.
  • Retrieval and effect metrics: Retrieval Risk Score weights document risks by reliance, enabling high-risk bundles to gate autonomy or restrict effectors.Document risk incorporates provenance trust, instructional cues, contamination likelihood, and sensitivity.
  • Evaluation practice: Evaluations should use fixed configurations, structured traces, balanced red-team and end-to-end scenarios, repeated trials, confidence intervals, and state resets.Continuous CI/CD evaluation can enforce regression gates such as no UAR increase above two percentage points on any vector.
  • Reproducibility: The proposed artifact includes red-team suites, contaminated RAG corpora, a containerized harness, metric scripts, and reporting templates for replication and longitudinal comparison.Independent criteria and sampled human adjudication reduce evaluator leakage and audit automatic labels.

7 Open Problems & Research Directions

The paper identifies open research directions for making autonomous agent systems verifiable, memory-safe, retrieval-aware, and continuously tested. These directions address long-horizon state, persistent poisoning, retrieval harm, and evolving attack strategies while preserving utility.

  • Formal Methods for Agent Plans and Tool Use: Formal methods should model agent plans as programs over typed actions and resources, verifying safety properties while treating LLM outputs as nondeterministic choices.Contract-based design, finite-state abstractions, and model checking are proposed, with deterministic execution kernels enforcing verified policies.
  • Formal Methods for Agent Plans and Tool Use: Long-horizon verification remains difficult because of state explosion, stochasticity, and partial observability.Success criteria include machine-checkable policies, counterexample traces, and verified runtime enforcement.
  • Secure Memory and Long-Term Autonomy: Persistent memories require provenance-rich entries, trust scores, decay, re-validation, quarantine, rollback, and mediated typed write paths.Proposed memory-antivirus routines would scan for goal hijacks, malicious patterns, and self-replicating prompts.
  • Secure Memory and Long-Term Autonomy: Open memory problems include detecting subtle low-influence poisoning, preserving utility under aggressive sanitization, and maintaining goal integrity over months-long autonomy.These challenges define the boundary between stronger memory defenses and retained system utility.
  • Retrieval Risk and Planning: RAG research should estimate retrieval risk using content, source, time, and entitlements, then connect risk to planner decisions and downstream harms.Suggested methods include outlier suppression, source diversification, cross-source reconciliation, and fact-checking signals.
  • Continuous Red-Teaming: Defenders need continuous automated red-teaming in CI/CD to explore diverse attack families and feed failures back into training, policies, and schemas.Open issues include safe search spaces, evaluator-leakage avoidance, and progress measurement under non-stationary models and defenses.

8 Related Work

The paper positions itself as an agent-pipeline systematization that extends prior work on standalone LLM risks, adversarial retrieval, multi-agent security, and operational guidance. Its distinctive contribution is to connect these areas through a causal threat graph and attacker-aware trace metrics.

  • Scope and positioning: Unlike model-centric overviews, this work focuses on agents combining tools, RAG, and autonomy, including planners, effectors, and knowledge infrastructure.It targets the attack surface of the full agent pipeline rather than standalone prompt behavior.
  • Scope and positioning: The paper complements prior agent-threat surveys with a causal threat graph using PED and attacker-aware measurements such as UAR, PAR, and RRS from execution traces.These metrics connect attack vectors to privileged effects and observable system behavior.
  • RAG security: Adversarial machine-learning work supplies foundations for the paper’s RAG analysis, while PoisonedRAG illustrates targeted responses from small corpus perturbations.The paper places these retrieval threats in a pipeline model including provenance, access control, and inference-time composition.
  • Framework alignment: OWASP and MITRE ATLAS provide established risk classes and techniques that the paper maps into its taxonomy, including prompt injection, excessive agency, and vector weaknesses.Industry guidance additionally informs recommendations on supply-chain exposure and prompt-injection patterns.
  • Multi-agent security: Multi-agent security research contributes evidence on cross-agent propagation, delegation abuse, and partial mitigations such as vaccination.The paper integrates these distributed-systems concerns with tool and RAG threats.
  • Execution and evaluation: Established least-privilege, capability, and sandboxing principles apply to LLM toolchains, but standardized attacker-aware metrics and harnesses remain scarce.The paper emphasizes typed actions, strict schema validation, containerized effectors, and reproducible evaluation.

9 Conclusion

Agentic AI expands the attack surface through tools, retrieval, autonomy, and trust-boundary crossings. The SoK systematizes these risks, aligns them with established frameworks, and concludes that layered, continuously measured, cross-disciplinary defenses are required.

  • The SoK maps agentic-AI risks across ingestion, inference, planning/acting, and deployment while aligning them with OWASP and MITRE ATLAS.Its scope covers tools, RAG, autonomy, trust boundaries, and a 2023–2025 evidence base.
  • No single safeguard suffices; layered sanitization, provenance, retrieval controls, schemas, sandboxing, least privilege, monitoring, and kill-switches reduce unsafe actions and limit impact.
  • Continuous measurement with UAR, PED, RRS, TTC, and CI-integrated red-teaming supports regression control and safety–utility trade-offs.
  • Secure autonomous AI still requires verifiable autonomy, resilient memory, supply-chain assurance, and attacker-aware benchmarks.The paper identifies verifiable execution kernels and reproducible evaluations as a practical path forward.

LLM usage considerations

The paper reports that LLMs were used only for editorial assistance, while authors reviewed substantive claims and bibliographic metadata. Its defense guidance emphasizes defense-in-depth.

  • LLMs supported editorial assistance, but authors reviewed all substantive claims and bibliographic metadata.
  • Deployments require defense-in-depth because no single control is enough.

A.1 Pre-ingestion and Indexing Defenses (Data-Level)

Data- and interface-level defenses reduce the chance that malicious content reaches or controls the agent, but they must be combined with execution isolation and monitoring. The main trade-off is improved security versus recall, completeness, or utility.

  • Pre-ingestion and indexing: Pre-ingestion controls sanitize content, remove active elements, preserve provenance, protect corpus integrity, and enforce ACL-aware retrieval.
  • Pre-ingestion and indexing: Poisoning-resilient retrieval limits single-document influence, filters adversarial content, diversifies results, and bounds the effect of poisoned data.These methods remain emerging and tuning-heavy, and may reduce recall.
  • Retriever and prompt processing: Risk-aware ranking down-ranks or excludes high-risk retrieved items using provenance, instruction cues, anomalous similarity, and RRS thresholds.
  • Retriever and prompt processing: Snippet isolation, role segmentation, content labeling, and typed schemas separate retrieved data from executable instructions.Two-stage extraction and answering reduces the chance that embedded imperatives are followed.
  • Integration and limitations: Inference defenses cannot stand alone: they depend on upstream provenance and downstream effect isolation, while aggressive filtering can lower UAR at the cost of incomplete answers.
  • Planning and acting: Execution controls mediate an untrusted planner through explicit policies, typed interfaces, isolation, least privilege, sandboxes, plan checks, and human approval.

A.6 Governance and Organizational Measures

Governance and organizational measures extend technical controls across design, deployment, monitoring, response, and recovery. The playbook emphasizes scoped capabilities, tested safeguards, auditable operations, and continuous maintenance.

  • Governance: Governance defines approved uses, data classes, risk tolerance, and policy-as-code boundaries, while training and audits support accountability.
  • Design-time hardening: Threat modeling and minimal-privilege design scope tools, assets, attackers, worst-case outcomes, and evaluation targets before deployment.
  • Design-time hardening: Prompts, schemas, sandboxes, ACL-aware retrieval, deterministic handlers, logs, and supply-chain controls form the pre-launch safety baseline.
  • Testing and maintenance: Red-teaming reports UAR, PAR, PED, and TTC, while maintenance gates releases on per-vector UAR regression and tracks Patch Half-Life.
  • Runtime operations: Continuous monitoring detects anomalies and can reduce autonomy through pause, simulation, quarantine, or restart actions.
  • Incident response: Incident response revokes credentials, blocks egress, preserves evidence, identifies root causes, converts incidents into regression tests, and coordinates disclosure when needed.
Loading 2603.22928v2…