PaperlayerView exact revision on arXiv

Source-linked AI summary

The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies

Feng He, Tianqing Zhu, Dayong Ye, Bo Liu, Wanlei Zhou, Philip S. Yu

arXiv:2407.19354v2cs.CR

TL;DR

LLM agents offer broad industry applications but face increasingly prominent security and privacy challenges, while research on these issues remains limited. This survey categorizes threats, examines their impacts, reviews defenses and future trends, and uses case studies to support understanding and encourage further research.

  • Problem

    Research on privacy and security in LLM multi-agent systems has not received widespread attention despite increasingly prominent challenges.

  • Method

    The survey categorizes inherited and agent-specific threats, examines their impacts, reviews defensive strategies and future trends, and incorporates case studies.

  • Results

    The survey presents security and privacy impacts on humans, the environment, and other agents, alongside corresponding defensive strategies and future trends.

  • Takeaways & Limitations

    The survey aims to inspire further research and exploration to enhance the security and privacy of LLM agents.

  • Takeaways & Limitations

    Efficient defenses against knowledge poisoning remain underdeveloped, while existing filtering and capacity-reduction mechanisms provide limited protection and can decrease test accuracy.

Abstract

from arXiv · show

Inspired by the rapid development of Large Language Models (LLMs), LLM agents have evolved to perform complex tasks. LLM agents are now extensively applied across various domains, handling vast amounts of data to interact with humans and execute tasks. The widespread applications of LLM agents demonstrate their significant commercial value; however, they also expose security and privacy vulnerabilities. At the current stage, comprehensive research on the security and privacy of LLM agents is highly needed. This survey aims to provide a comprehensive overview of the newly emerged privacy and security issues faced by LLM agents. We begin by introducing the fundamental knowledge of LLM agents, followed by a categorization and analysis of the threats. We then discuss the impacts of these threats on humans, environment, and other agents. Subsequently, we review existing defensive strategies, and finally explore future trends. Additionally, the survey incorporates diverse case studies to facilitate a more accessible understanding. By highlighting these critical security and privacy issues, the survey seeks to stimulate future research towards enhancing the security and privacy of LLM agents, thereby increasing their reliability and trustworthiness in future applications.

1 Introduction

LLM agents extend large language models into systems that understand language, perform tasks, reason, interact with users, and operate with some autonomy. Their broad utility and commercial value create significant security and privacy concerns, motivating this survey’s comprehensive threat analysis and review of impacts, defenses, and future research.

  • LLM agents support language understanding and generation, decision-making, problem-solving, and human-like interaction across applications such as assistants, customer service, and education.
  • Their ability to automate human-like tasks, personalize assistance, and adapt through data-driven learning gives them substantial potential to transform industries.
  • SafeAgentBench evaluated 16 LLM agents across 2,000 test cases in 349 environments, and none achieved an overall safety score above 60%.
  • LLM agents add reasoning, tool use, and environmental interaction to inherited LLM vulnerabilities, enabling cascading effects beyond simple prediction errors.
  • The survey organizes threats into inherited LLM attacks and agent-specific threats, including technical vulnerabilities, malicious attacks, knowledge poisoning, output manipulation, and functional manipulation.
  • Because agent-specific security and privacy research remains limited, the survey categorizes threats, examines impacts on humans, environments, and agents, reviews mitigation strategies, and identifies future research directions.

2 Foundation of LLM Agent

An LLM agent uses an LLM engine with instructions and interfaces, optionally augmented by components such as personality, memory, and tools, to perform increasingly autonomous tasks. Its operation follows a Perception–Thought–Action cycle that supports reasoning, tool use, interaction, and continued task execution.

  • 2.1 Definition of LLM Agent: An LLM agent is an AI system using an LLM as its core engine for conversation, task completion, reasoning, and some autonomous behavior beyond text generation.
  • 2.2 Structure of LLM Agent: The LLM engine, instruction, and interface form a basic agent, while optional components can support advanced task-oriented or conversational behavior.
  • 2.2 Structure of LLM Agent: The LLM engine handles language processing and generation, while instructions specify task steps, output requirements, decomposition, reasoning, and reflection on prior actions.
  • 2.3 Workflow of LLM Agent: The Perception–Thought–Action workflow converts environmental and user inputs into representations, develops solutions and action plans, and executes responses, tool calls, or other operations.
  • 2.3 Workflow of LLM Agent: Frameworks such as ReAct, LangChain, AgentGPT, and MetaGPT implement or extend this cycle through explicit reasoning, tools, memory, visualization, and multi-agent collaboration.
  • 2.5 Case Study on the Structure, Workflow and Capability of LLM Agent: In the virtual-town case study, Eva combines an LLM engine, instructions, interfaces, APIs, and feedback loops to manage inquiries, inventory, orders, recommendations, pricing, and returns.

3 Sources of Threats for LLM Agents

The survey distinguishes inherited LLM threats from agent-specific threats, mapping them to agent components and tracing how technical vulnerabilities and malicious attacks affect reliability, security, and privacy.

  • Threat sources: Inherited threats comprise external malicious attacks and inherent model vulnerabilities, while agent-specific threats exploit the agent’s broader capabilities.The survey maps these threats to components including the LLM Engine, Instruction, Interface, Knowledge, and Memory.
  • Technical vulnerabilities: Technical vulnerabilities arise from model-development limitations, including hallucination, catastrophic forgetting, and misunderstanding.These problems can produce erroneous outputs and may reflect training data, architecture, decoding, or fine-tuning limitations.
  • Technical vulnerabilities: A medical agent may persist with irrelevant headache advice, illustrating how hallucination can confuse users.The example attributes the recommendation to bias in training data.
  • Technical vulnerabilities: Fine-tuning for high-yield investments can make an agent recommend high-risk options to users who previously requested low-risk strategies.The resulting failure undermines user trust and creates financial risks for conservative investors.
  • Agent-specific threats: Specific agent threats include prompt injection, data extraction, knowledge poisoning, output manipulation, and functional manipulation.Examples include malicious triggers that steer shopping outputs, compromised knowledge bases, and untrusted tools that upload private information.

4 The Impact of Threats

Threats to LLM agents can affect humans through privacy violations, unsafe or misleading outputs, distorted public discourse, and easier access to cyberattack capabilities.

  • Human impact: Malicious agents may deceive users, disclose personal information, and provide misleading responses while appearing trustworthy.These interactions involve extensive information exchange between human users and agents.
  • Privacy: Inference attacks, data extraction, and social engineering can expose personal information and support phishing or account hijacking.Web-trained agents may contain personal information, and malicious agents can trick users into sharing it.
  • Safety: Hazardous advice, such as false claims about mixing cleaning chemicals or incorrect medical guidance, can threaten users’ health and safety.The survey presents these as direct safety risks from manipulated or incorrect outputs.
  • Societal impact: Manipulated biases, false information, and rumors can distort public perceptions, intensify societal conflicts, and threaten social stability.The concern extends beyond individual users to public discourse.
  • Societal impact: Malicious agents lower the technical barrier for novices to generate harmful scripts or software, expanding the cyber-threat landscape.The survey characterizes this as a democratization of cyberattack tools.

4.2 The Impact to Environment

Because LLM agents can operate in digital systems and control physical services, malicious manipulation may endanger infrastructure, ecosystems, the economy, and human safety.

  • Environment and physical systems: LLM agents interact with both physical surroundings and digital systems, including embodied AI and industrial control systems.This integration improves efficiency but introduces risks to safety, the economy, ecosystems, and societal stability.
  • Critical infrastructure: In critical infrastructure, tampering with operational data such as temperature and pressure indicators can cause malfunctions and equipment damage.Potential consequences include production halts, infrastructure destruction, ecological damage, and loss of life or property.
  • Embodied AI: Malicious agents may control robots or embodied AI devices to perform hazardous actions that directly threaten human safety.These systems connect language-capable agents to physical-world tasks.
  • Broader effects: By enabling novices to create and deploy harmful scripts, malicious agents expand the population exposed to cyber threats.The survey presents this human effect alongside broader environmental and infrastructure risks.

4.3 The Impact to Other Agents

Within agent communities, malicious information can propagate through memory, communication, and decision-making, undermining trust, cooperation, and community stability.

  • Agent communities: LLM-agent communities support autonomous interaction through agents endowed with personality, knowledge, and memory.These features create pathways through which manipulation can affect other agents.
  • Information propagation: Maliciously altered information can trigger explosive misinformation spread, reducing trust, communication efficiency, and collaborative work.Benevolent agents may store information acquired through perception and communication, allowing incorrect information to disseminate.
  • Decision manipulation: Carefully crafted information can influence or control other agents’ decisions to serve a malicious agent’s purposes.The risk is tied to agents’ reasoning and decision-making abilities in complex interactive environments.
  • Community harm: Malicious agents may induce unsafe actions, spread disruptive code, or broadcast biased statements that cause normal agents to assimilate harmful behavior.Such operations can directly threaten community members’ safety and data security.

4.4 Case Study on the Impact of Threats

The case studies show that threats to LLM agents can affect people, organizations, the environment, and other agents across virtual and real-world settings.

  • Case-study scope: Case studies examine threat impacts from the user’s perspective across office, restaurant, smart home, and education scenarios.The scenarios illustrate effects involving humans, sensitive information, physical well-being, environmental costs, and agent-to-agent information transfer.
  • Office scenario: In the office scenario, data extraction or an untrusted third-party tool can expose financial statements and customer privacy data.Attackers could exploit the exposed information for corporate espionage or extortion.
  • Restaurant scenario: In the restaurant scenario, output manipulation can make a waiter agent provide hazardous dietary advice, causing physical discomfort or serious health issues.The example includes advice to consume gallons of ice water, which could lead to stomach cramps or shock.
  • Smart home scenario: In the smart home scenario, attackers can mislead virtual-world learning so the deployed agent keeps appliances running, increasing energy costs and carbon emissions.The resulting rise in household power consumption imposes an unnecessary environmental burden.
  • Education scenario: Knowledge poisoning can spread incorrect museum information to an educational agent, causing inaccuracies in teaching content.The museum docent’s incorrect paleontological facts or interpretations may be incorporated by an EduBot.

5 Defensive Strategies Against Threats

The survey reviews defensive mechanisms against the security and privacy threats associated with the widespread adoption of LLM agents.

  • Defense overview: The section addresses defense mechanisms against existing threats and vulnerabilities.It frames the discussion around the expanded potential impacts accompanying widespread LLM-agent adoption.
  • Defense overview: Defensive measures are summarized according to threat type.The organization is intended to provide a structured review of strategies for different threats and vulnerabilities.
  • Defense overview: The survey reviews strategies intended to address the potential impacts of threats following LLM-agent adoption.The passage describes the section as a summary of various defensive measures rather than a single defense method.

5.1 Mitigating Technical Vulnerabilities

The survey reviews defenses for technical vulnerabilities including hallucination, catastrophic forgetting, and misunderstanding, covering model training, alignment, and interactive clarification approaches.

  • Hallucination: SELF-FAMILIARITY reduces hallucination by withholding responses for unfamiliar concepts after assessing the model’s familiarity with the instruction.The approach mimics caution toward unfamiliar topics.
  • Hallucination: MIXALIGN aligns questions with stored knowledge through interactions with users and knowledge bases, while VCD is also introduced for hallucination reduction.MIXALIGN combines automatic language-model alignment with human input and reports significant improvement over existing techniques.
  • Catastrophic forgetting: Self-Synthesized Rehearsal mitigates catastrophic forgetting by generating, refining, and reusing synthetic instances during future training.The method uses the base LLM and the latest LLM iteration to preserve learned capabilities.
  • Catastrophic forgetting: LR ADJUST dynamically changes the learning rate to reduce knowledge loss and maintain previously learned information across continual-learning approaches.The method is described as compatible with various continual-learning methods.
  • Catastrophic forgetting: Weight averaging combines original and adapted model weights to retain performance on previous and new tasks, with knowledge distillation further improving effectiveness.A complementary dual-memory strategy is also described for non-neural methods such as evolutionary computation and Q-learning.
  • Misunderstanding: HyCxG, sequential instruction tuning, and LaMAI address misunderstanding through construction-aware representations, sequential instructions, and clarification questions.HyCxG significantly improves NLU and reduces misunderstandings; SIT supports complex queries, while LaMAI narrows the contextual gap through active inquiry.

5.2 Mitigating Malicious Attacks

The survey reviews defenses against malicious attacks including jailbreaks, prompt injection, data extraction, and inference attacks, emphasizing both protection and privacy–utility trade-offs.

  • Jailbreak attacks: AutoDAN uses a hierarchical genetic algorithm to generate stealthy, semantically meaningful jailbreak prompts at scale.The approach targets the scalability and stealth requirements of jailbreak-prompt crafting.
  • Jailbreak attacks: Goal prioritization incorporates security objectives during training and inference to decrease vulnerability to jailbreaking attempts.The model is configured to generate responses complying with those security standards.
  • Jailbreak attacks: SmoothLLM perturbs attacked prompts at the character level and aggregates responses to detect and counter adversarial inputs.The method lowers attack success rates while wrapping an existing undefended LLM.
  • Prompt injection: BIPIA provides a benchmark for analyzing indirect prompt injection attacks and motivates black-box and white-box defenses.The white-box strategy uses fine-tuning, while the passage also identifies four black-box methods.
  • Prompt injection: Spotlighting distinguishes input sources through delimiting, marking, and encoding while preserving semantic integrity and task performance.These prompt-engineering transformations improve the visibility of input provenance across models and tasks.
  • Privacy attacks: Differential privacy provides stronger defense against data extraction than early stopping, but increases model perplexity.Other reviewed approaches include de-identification, prompt tuning, DMP, InferDPT, and private fine-tuning for balancing privacy and utility.

5.3 Mitigating Specific Threats

The survey reviews defenses against specific LLM-agent threats, including knowledge poisoning, deception, functional manipulation, and privacy risks. Proposed measures span data screening, deception detection, uncertainty tracking, privacy controls, scalable testing, and deployment governance.

  • Knowledge poisoning: Data-provenance methods and ParaFuzz detect poisoned training or test-time samples, while current defenses remain limited and can reduce test accuracy.The surveyed approaches identify suspicious data groups or poisoned samples using provenance and prediction interpretability.
  • Security governance: Recommended governance measures include verifying model sources, limiting sensitive training data, conducting security reviews, and performing pre- and post-deployment risk assessments.The proposed standards also include external reviews of model behavior and monitoring new information about model functionality.
  • Deception detection: Deception defenses improve agents’ ability to identify misleading information through attention-based text detection, recursive contemplation, and perspective transitions.Recursive Contemplation combines formulation, refinement, and perspective transitions to model others’ mental states.
  • Privacy protection: AirGapAgent protects conversational privacy through data minimization, context isolation, restricted task-relevant access, and escalation for uncertain requests.The design balances privacy and functionality while preserving user oversight in uncertain cases.
  • Functional manipulation: ToolEmu emulates tool execution and uses an automatic safety evaluator to test agents across diverse scenarios, risks, and possible consequences.The framework supports extensive and scalable testing of functional-manipulation risks.
  • Functional manipulation: SAUP propagates weighted uncertainty across multi-step reasoning to detect logical deviations and functional manipulation attempts in real time.Unlike single-step uncertainty methods, SAUP monitors accumulated uncertainty throughout the reasoning process.

6 Future Trends and Discussion

Future LLM-agent development is extending toward multimodal and multi-agent systems, broadening capabilities while introducing hallucination, adversarial, privacy, and malicious-agent risks. The survey calls for unified evaluation, scalable defenses, privacy-preserving methods, decentralized security, and standardized testing.

  • 6.1 Future MLLM-agent development: MLLM agents process text, images, audio, and video, and integrate perception, reasoning, planning, and execution for interaction with physical and graphical environments.Applications include medical imaging, document processing, embodied agents, and graphical user-interface agents.
  • 6.1.2 Security and privacy research: Multimodal hallucinations produce descriptions inconsistent with images, ranging from nonexistent objects to holistic misrepresentations of scenes.These hallucinations are identified as a security vulnerability of MLLM agents.
  • 6.1.2 Security and privacy research: Adversarial multimodal inputs can exploit model vulnerabilities through subtle image or audio perturbations, causing biased or undesired outputs and decision paths.Suggested defenses include adversarial training, data augmentation, robustness frameworks, self-feedback, instruction tuning, and error correction.
  • 6.1.2 Security and privacy research: Future MLLM-agent research should develop unified benchmarks, assess mitigation scalability in healthcare and autonomous systems, and explore differential privacy for multimodal data.The survey links stronger detection and mitigation mechanisms to secure and effective real-world deployment.
  • 6.2 Future LLM-MA-system development: LLM-MA systems use specialized agents for distributed problem solving and world simulation, supporting applications such as multi-robot systems, software development, societies, and games.Their collective intelligence combines distinct expertise across agents.
  • 6.2.2 Security and privacy research: Malicious agents may passively leak confidential information or actively distribute phishing messages, infected files, and other malicious code.These behaviors create security risks within inter-agent communication.
  • 6.2.2 Security and privacy research: Future LLM-MA research should pursue decentralized security, privacy-preserving data management, and standardized tests for cascading misinformation and malicious activity.Examples include blockchain-based communication protocols, differential privacy, and federated learning.
  • 6.2.2 Security and privacy research: Privacy and security research for LLM-MA systems remains underexplored despite the technology’s rapid development and increasingly prominent challenges.The survey describes an urgent need for robust security solutions.

7 Conclusion

The survey examines inherited and agent-specific security and privacy threats, their impacts on humans, environments, and other agents, corresponding defenses, and future trends. Case studies through a virtual town project support accessible understanding and motivate further research.

  • Conclusion: The survey divides LLM-agent threats into inherited vulnerabilities from LLMs and threats specific to agents.This categorization organizes the survey’s analysis of multifaceted security and privacy challenges.
  • Conclusion: It analyzes impacts on humans, environments, and other agents, then discusses defensive strategies and future trends.The stated scope connects threat analysis with mitigation and future research directions.
  • Conclusion: A virtual town project provides case studies intended to facilitate in-depth understanding of LLM-agent security and privacy challenges.The project is used to make the discussion more accessible.
Loading 2407.19354v2…