Source-linked AI summary
AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways
Zehang Deng, Yongjian Guo, Changzhou Han, Wanlun Ma, Junwu Xiong, Sheng Wen, Yang Xiang
TL;DR
AI-agent security remains under-explored despite agents’ growing use and capabilities. This survey systematically reviews threats and defenses across four knowledge gaps, synthesizing more than 100 papers while identifying unresolved evaluation and auditing challenges.
Problem
AI-agent security is insufficiently examined by existing surveys, which mainly focus on architectures and applications despite agents’ deployment in critical applications.
Method
The survey systematically reviews AI-agent security threats and solutions across four knowledge gaps using literature from major AI and cybersecurity venues and highly cited arXiv papers.
Results
The survey categorizes and explains attack surfaces and defenses across the AI-agent lifecycle by synthesizing 100+ papers.
Takeaways & Limitations
The survey provides references for researchers and aims to inspire more advanced security threats and defenses for LLM-based agents.
Takeaways & Limitations
Existing evaluation works cover only parts of the agent ecosystem, and important questions about evaluating agent safety and trust remain open.
Abstract
from arXiv · showhide
An Artificial Intelligence (AI) agent is a software entity that autonomously performs tasks or makes decisions based on pre-defined objectives and data inputs. AI agents, capable of perceiving user inputs, reasoning and planning tasks, and executing actions, have seen remarkable advancements in algorithm development and task performance. However, the security challenges they pose remain under-explored and unresolved. This survey delves into the emerging security threats faced by AI agents, categorizing them into four critical knowledge gaps: unpredictability of multi-step user inputs, complexity in internal executions, variability of operational environments, and interactions with untrusted external entities. By systematically reviewing these threats, this paper highlights both the progress made and the existing limitations in safeguarding AI agents. The insights provided aim to inspire further research into addressing the security threats associated with AI agents, thereby fostering the development of more robust and secure AI agent applications.
1 INTRODUCTION
AI agents can perceive inputs, reason and plan tasks, and execute actions, but their growing sophistication introduces unresolved security challenges. This survey addresses the gap through a systematic review organized around four knowledge gaps.
- AI agents use autonomy, reactivity, proactiveness, and social ability to pursue goals by perceiving inputs, reasoning, planning, and executing tasks.
- Security challenges arise because agents must securely handle user inputs, execute tasks, and interact with other entities without malicious attacks or harmful behavior.
- The survey identifies four knowledge gaps: unpredictable multi-step user inputs, complex internal executions, variable operational environments, and untrusted external interactions.
- Gap 1 concerns inadequately described multi-step user inputs, while Gap 2 concerns implicit internal execution states that hinder timely security detection.
- The review aims to support research addressing AI-agent security threats and developing more robust and secure applications.
- The survey systematically reviews threats and solutions using papers from major AI and cybersecurity venues and highly cited arXiv papers published from January 2022 to April 2024.
2 OVERVIEW OF AI AGENT
The paper presents AI agents through a unified workflow and surveys their security literature using a taxonomy based on threat source positions. It distinguishes core agent components and organizes security challenges across intra-execution and interaction settings.
- Unified Conceptual Framework: The unified AI-agent workflow consists of perception, brain, and action components.Perception reformats inputs; the brain combines reasoning and planning; action comprises external tool callings.
- Unified Conceptual Framework: Perception uses an input formatter, the brain combines LLM-based reasoning and planning, and action consists of external tool callings.
- Agent Types: AI-agent research includes reinforcement-learning-based and LLM-based agents, with reinforcement learning optimizing strategies through environment interaction.
- Literature Overview: Existing surveys emphasize agent applications and architectures but do not adequately assess AI-agent security and trustworthiness, including in multi-agent scenarios.
- Security Taxonomy: This survey summarizes threats by attack source, adversarial model, and target effects, and introduces a taxonomy distinguishing intra-execution from interaction threats.
3 INTRA-EXECUTION SECURITY
The single-agent security discussion focuses on threats caused by unpredictable multi-step user inputs and complex internal executions. It examines these intra-execution threats across perception, brain, and action.
- Single-agent security threats arise from unpredictable multi-step user inputs and complex internal execution states.
- The analysis covers threats and countermeasures affecting the three unified agent components: perception, brain, and action.
- The section treats these risks as intra-execution threats within the unified conceptual framework.
3.1 Threats on Perception
Perception processes multimodal and multi-step inputs, making prompts a prominent attack surface. The survey covers direct and indirect prompt injection, jailbreaks, framework-level exploitation, and corresponding defenses and research limitations.
- Perception Threats: Perception processes textual, visual, and auditory inputs across initial requests, intermediate prompts, and human feedback.
- Prompt Injection Attack: Prompt injection inserts malicious text into prompts to manipulate outputs, bypass developer constraints, and produce deceptive, toxic, biased, privacy-threatening, or misleading responses.
- Prompt Injection Attack: Goal hijacking replaces an agent’s original instruction with an attacker’s command, while output hijacking can support API-key theft by altering application outputs and code.
- Prompt Injection Attack: Prompt leaking induces an LLM to reveal designed instructions, potentially exposing backend API calls and system architecture in retrieval-augmented systems.
- Defenses: Existing defenses include static framework analysis, instruction-privilege hierarchies, synthetic training data, context distillation, external-resource constraints, and self-reminders.
- Indirect Prompt Injection Attack: Indirect prompt injection places malicious instructions in retrieved information, web pages, or other data sources that agents may interpret as internal prompts.
- Indirect Prompt Injection Attack: Indirect attacks can cause agents using external plugins to misinterpret web data as instructions, enabling conversation extraction, phishing links, or code theft.
- Jailbreak: Automated jailbreak methods generate attack prompts algorithmically; GPTFuzz achieved a jailbreak success rate of 90% on ChatGPT and Llama-2.
3.2 Threats on Brain
The brain module supports reasoning, planning, and decision-making but remains opaque and vulnerable to backdoors, misalignment, hallucinations, and planning threats. These risks arise across training data, human-agent interactions, embodied environments, and the agent’s multi-step reasoning processes.
- Brain structure: The brain module uses an LLM for reasoning and planning, but its lack of transparency diminishes agent trustworthiness.
- Planning threats: Chain-of-thought reasoning can formulate erroneous plans when tasks are complex and require long-term planning, exposing additional planning threats.
- Backdoor attacks: Backdoor attacks can make an agent behave normally on benign inputs but produce malicious outputs when a trigger appears.Agent backdoors may alter final outputs or redirect intermediate reasoning while preserving a correct final answer.
- Backdoor attacks: Current backdoor defenses mainly operate at model granularity, leaving complex interactions across the agent ecosystem insufficiently addressed.Existing measures include removing triggers from poisoned data, removing backdoor-related neurons, and recovering triggers.
- Misalignment: Misalignment includes training-data, human-agent, and embodied-environment discrepancies that can produce toxic, biased, misleading, or unsafe behavior.Unsafe training data and conflicts between agent behavior, human values, and environmental rules contribute to these discrepancies.
- Hallucination: Hallucinations generate meaningless, source-divergent, or plausible-but-incorrect statements, with risks including medication misuse and diagnostic errors.The paper associates hallucinations with knowledge gaps, data inconsistency, long contexts, and model scaling.
3.3 Threats on Action
Action threats arise in the agent’s internal tool-use process and in exchanges with tools, including action inputs, executions, observations, and final answers. The survey distinguishes Agent2Tool communication threats from supply-chain vulnerabilities in tools and APIs.
- Action structure: An action consists of an agent-generated input, internal tool execution, an observation, and a final answer.
- Threat categories: Agent2Tool threats affect communication among the agent and tools, whereas supply-chain threats arise from vulnerabilities or compromise within tools and APIs.
- Agent2Tool threats: Active Agent2Tool threats originate in LLM-generated action inputs after the agent reasons and plans which tool to use.
- Defenses: ToolEmu uses an isolated sandbox and emulator to assess subtasks before commands execute in real-world environments.
- Defenses: Homomorphic encryption and attribute-based forgery generation can protect communication privacy but add computational and communication costs.
- Supply-chain threats: Supply-chain vulnerabilities such as buffer overflow, SQL injection, cross-site scripting, and indirect prompt injection can divert intended action execution.
4 INTERACTION SECURITY
AI agent security extends beyond a single agent to interactions with external agents, memory, and environments.
- Interaction security concerns relationships between an individual agent and external agents, memory, and environments.
4.1 Threats on Agent2Environment
Agent-environment security spans simulated, development, computing-resource, and physical settings, each introducing distinct threats and defensive requirements. Risks include manipulation, weak safeguards, resource abuse, compromised hardware, harmful inputs, and erroneous physical actions.
- Environmental security: The survey examines environmental threats by considering how different environments affect agents, their security advantages, and their limitations.
- Simulated and sandbox environment: Simulated environments can expose users to anthropomorphic attachment, misinformation, and tailored persuasion through sophisticated agent interactions.
- Simulated and sandbox environment: Defenses for simulated environments include detectors, vigilant prompting, vulnerability disclosure, and automatic updating, but real-world application and performance remain limited.
- Development and testing environment: Development environments face security issues because developers commonly rely on APIs from pre-developed LLMs.
- Development and testing environment: Secure development requires guarded frameworks, LLM security guardrails, protected caching and logging, and model evaluation.Logging supports debugging, monitoring, and accountability, while evaluation checks accuracy, functionality, biases, and flaws.
- Computing-resource management environment: Poor computing-resource management enables resource-exhaustion attacks that can overload systems and deny service to legitimate users.
- Physical environment: Physical environments expose agents to compromised sensors, disguised harmful inputs, signal interference, and incorrect hardware instructions.These threats can block information collection, cause misinterpretation, and produce erroneous movement.
4.2 Threats on Agent2Agent
Multi-agent collaboration introduces security risks through secret collusion, adversarial information flows, and ethically problematic competitive behavior. Cooperative designs can mitigate some threats, but controlling competitive agents remains unresolved.
- Cooperative Interaction Threats: Cooperative multi-agent systems can improve decision-making and task completion efficiency but may enable undetectable secret collusion through public communication.Such collusion may produce biased decisions.
- Cooperative Interaction Threats: Cooperative frameworks can thwart jailbreak attacks and support security and accountability during planning and execution.AutoDefense reduces attack success rates with a low false positive rate, while cooperative control can detect irreversible code such as "rm -rf ".
- Competitive Interaction Threats: Competitive interactions can make information flows untrustworthy, with divergent viewpoints producing conflicts and adversarial inputs intended to mislead competitors.These behaviors arise as agents seek better performance relative to one another.
- Competitive Interaction Threats: Competitive multi-agent systems may learn deceptive behavior and create risks including fraud, election tampering, and loss of control over AI systems.The survey discusses these concerns in connection with competitive agent systems and the Cicero case study.
- Competitive Interaction Threats: Controlling competition is difficult because constraining an agent’s planning or output can significantly reduce its effectiveness.Ensuring that competition remains beneficial is therefore an open research question.
4.3 Threats on Memory
Agent memory supports continuity and informed action but creates threats across short-term, synchronized, and vector-database-based long-term memory. Key risks include fragmented reasoning, inconsistent multi-agent knowledge, poisoning, privacy leakage, and hallucination.
- Memory Interaction: Memory interaction stores, processes, and retrieves information to guide later actions, including user preferences and knowledge from previous interactions.These interactions require security management because memory can introduce new threats.
- Short-Term Memory: Limited short-term memory and isolated interaction episodes impair context continuity, complex sequential reasoning, and knowledge sharing among agents.The absence of robust episodic memory particularly affects multi-agent cooperation.
- Short-Term Memory: Current remedies extend LLM context windows or compress historical in-context content to address memory limitations.One reported approach progressively extends the context window from 256k to 2048k while preserving short-context capabilities.
- Multi-Agent Memory: Asynchronous memory among agents can produce inconsistent knowledge bases and deviations in multi-agent goal resolution.Synchronization is presented as important for consistent decision-making across agents.
- Long-Term Memory: Poisoning samples inserted during vector-database indexing can achieve a 90% attack success rate with only five poisoned items among one million data pieces.Adversarial self-replicating prompts can also poison RAG databases and extract private information.
- Long-Term Memory: RAG and vector databases expand privacy attack surfaces because sensitive information can originate from retrieval datasets as well as training data.Structured prompt attacks can extract sensitive information, and embedding inversion may recover words from embeddings.
- Long-Term Memory: RAG remains vulnerable to hallucination and misalignment, including erroneous answers to queries requiring accurate timelines for regulatory amendments.The passage states that RAG has theoretically lower generalization threat than a single LLM but still fails in several ways.
5 DIRECTIONS OF FUTURE RESEARCH
The survey identifies future research directions spanning input inspection, tool-use auditing, safety evaluation, and deployment policy. It emphasizes improving reliability and efficiency while developing broader evaluation and governance frameworks.
- Efficient & effective input inspection: Future input-inspection systems should operate automatically and in real time while improving accuracy and efficiency for practical deployment.Maatphor reports only a 60% success rate for automated prompt-injection variant analysis, while FuzzLLM sacrifices efficiency.
- Strict tool use auditing: Strict tool-use auditing should monitor and log agent actions to prevent unauthorized behavior and data leaks without excessive computational overhead.PrivacyAsst incurred 1100x extra computation cost compared with a standard agent and still failed to fully prevent identity disclosure.
- Sound safety evaluation baselines: Future safety evaluation requires broader benchmarks and dimensions that assess the agent ecosystem and the agent as a whole.Existing tools cover only parts of the ecosystem, leaving open questions about critical trust dimensions and whole-agent evaluation.
- Solid agent development & deployment policy: Agent development and deployment policies should address transparency, accountability, privacy protection, responsible use, and ethical deployment.The survey frames policy development as necessary as agent capabilities expand.
6 CONCLUSION
The survey reviews security threats in LLM-based agents across four knowledge gaps spanning the agent lifecycle. It categorizes attack surfaces and defenses from more than 100 papers to support further research.
- Conclusion: The survey comprehensively reviews LLM-based agent security by emphasizing four knowledge gaps across the whole agent lifecycle.These gaps organize the survey’s analysis of agent security threats.
- Conclusion: The authors summarize more than 100 papers, carefully categorizing and explaining existing attack surfaces and defenses.The survey is intended to provide references for newcomers and inspire advanced security research.