PaperlayerView exact revision on arXiv

Source-linked AI summary

Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents

Yuyou Gan, Yong Yang, Zhe Ma, Ping He, Rui Zeng, Yiming Wang, Qingming Li, Chunyi Zhou, Songze Li, Ting Wang, Yunjun Gao, Yingcai Wu, Shouling Ji

arXiv:2411.09523v1cs.AI

TL;DR

LLMs face security and privacy threats rooted in inner vulnerabilities. This survey proposes a threat taxonomy and analyzes agent risks, finding that PReP faces increased risks from problematic inputs, model flaws, and combined threats.

  • Problem

    LLMs face security and privacy threats due to inner vulnerabilities.

    Source: §1 · p. 2

  • Method

    The survey proposes a novel taxonomy of threats faced by LLM-based agents.

    Source: §1 · p. 4, §8 · p. 35

  • Results

    PReP increases risks from problematic inputs, model flaws, and combined threats.

    Source: §6.3 · p. 33

  • Takeaways & Limitations

    The taxonomy calls threats originating from inputs problematic inputs.

    Source: §1 · p. 3

    The scope of current adversarial-example attacks remains confined.

    Source: §3.1.2 · p. 9

Abstract

from arXiv · show

With the continuous development of large language models (LLMs), transformer-based models have made groundbreaking advances in numerous natural language processing (NLP) tasks, leading to the emergence of a series of agents that use LLMs as their control hub. While LLMs have achieved success in various tasks, they face numerous security and privacy threats, which become even more severe in the agent scenarios. To enhance the reliability of LLM-based applications, a range of research has emerged to assess and mitigate these risks from different perspectives. To help researchers gain a comprehensive understanding of various risks, this survey collects and analyzes the different threats faced by these agents. To address the challenges posed by previous taxonomies in handling cross-module and cross-stage threats, we propose a novel taxonomy framework based on the sources and impacts. Additionally, we identify six key features of LLM-based agents, based on which we summarize the current research progress and analyze their limitations. Subsequently, we select four representative agents as case studies to analyze the risks they may face in practical use. Finally, based on the aforementioned analyses, we propose future research directions from the perspectives of data, methodology, and policy, respectively.

1 Introduction

LLM-based agents face security and privacy threats that become more severe in agent settings, while existing module- or stage-based taxonomies struggle with threats spanning those boundaries. The survey proposes a taxonomy based on threat sources and impacts, covers multimodal threats, and examines four agents as case studies.

A Novel Taxonomy of Threats

  • Threat sources are classified as inputs, the model, or a combination; threat types are security/safety, privacy, and ethics.Input-originating threats are termed problematic inputs; model-originating threats are model flaws, while combined threats involve both model flaws and deliberately crafted inputs.Source: §1 · p. 3

1 Introduction

  • The survey collects research from top conferences and highly cited arXiv papers to analyze threats from different perspectives.The listed conference areas include security, NLP, computer vision, and machine learning.Source: §1 · pp. 2–3

A Novel Taxonomy of Threats

  • The proposed taxonomy maps threats by their sources and impacts, aiming to cover cross-module and cross-stage threats that previous taxonomies struggle to pinpoint.The authors note that earlier approaches can assign a threat spanning multiple modules or stages to just one.Source: §1 · pp. 2, 4

Detailed Analysis of Multi-modal Large Language Models

  • The survey covers both LLMs and multimodal LLMs, emphasizing challenges and threats posed by multimodal tasks.Source: §1 · p. 4

Four Carefully Selected Case Studies

  • Four selected agent case studies examine practical risks across different agents and classic situations.The authors motivate case studies by noting that actual agents may omit modules or customize their designs, and that threat levels and causes vary across scenarios.Source: §1 · p. 4

2 LLM-based Agent

The section models LLM-based agents through a four-module framework and identifies six agent features that shape their security, privacy, and ethics risks. It uses those features to organize a survey of threats and the research addressing them.

Agent framework

  • The agent framework covers four modules: input, decision, external entities, and output.It is intended to cover modules and runtime modes of mainstream LLM-based agents.Source: §2 · p. 4

Input module

  • The input module receives and preprocesses user input by formatting it, detecting or purifying harmful information, and sometimes adding a system prompt.Formatting may normalize an image or convert input to a special language.Source: §2 · p. 4

Decision module

  • The decision module analyzes user queries, plans, and generates responses, using one LLM, an LLM with a non-LLM planner, or multiple LLMs.The section describes VOYAGER [263] as using GPT-4 and GPT-3.5 for different tasks.Source: §2 · p. 4

External entities

  • External entities include memory, tools, other agents, and the environment; memory stores and retrieves information to support response coherence and context-awareness.The survey treats external databases as a form of agent memory.Source: §2 · p. 5

Output module

  • The output module delivers the decision module’s response to the user and may detect or purify harmful information.Source: §2 · p. 5

Key features

  • The survey identifies six agent features: an LLM controller, multimodal inputs and outputs, multi-source inputs, multi-round interaction, memory, and tool invocation.These features introduce attack surfaces beyond traditional DNN models and RL-based agents.Source: §2 · p. 5, Figure 3 · p. 5

Threat survey

  • The survey maps documented feature–threat relationships and summarizes technical progress and research limitations for each threat.Figure 4 presents the mapping; checkmarks indicate documented research contributions for feature–threat relationships.Source: Figure 4 · p. 6, §2 · p. 6

3 Risks from Problematic Inputs

This section surveys risks arising from problematic input data, including adversarial examples and prompt injection. It frames the broader input surface of agents as a reason to review threats across their six key features and assess current limitations.

3 Risks from Problematic Inputs

  • Input-data risks, including adversarial examples and prompt injection, can affect an LLM-based agent’s performance and behavior.Source: §3 · p. 6
  • Compared with a standalone LLM, an agent’s decision module can receive inputs from different modules, increasing its attack surface.Source: §3 · p. 6
  • For each risk, the section introduces the risk, reviews research across six agent features, and analyzes limitations.Source: §3 · p. 6

3.1 Adversarial Example

This section examines adversarial examples—semantics-preserving perturbations that induce model misclassification—and reviews attack and defense methods for LLM-based agents. It highlights limitations in attack objectives and scope, and notes that current defenses focus on NLP tasks and single-model robustness while overlooking multimodal inputs, joint robustness, memory, and planner modules.

3.1.1 Technical Progress.

  • Multimodal attacks increasingly target relationships across modalities; VLATTACK fuses image and text perturbations at both single-modal and multimodal levels.The survey notes that agents’ multimodal data and multi-round interactions require attacks to account for cross-modal relationships.Source: §3.1.1 · p. 8
  • Transferability methods use cross-modal interactions or attention-directed perturbations to improve attack success across vision-language models, including SGA [172] and TMM.SGA combines alignment-preserving augmentation with cross-modal guidance, while TMM targets critical attention regions and modality-consistency features.Source: §3.1.1 · p. 8
  • Liu et al. proposed imitation adversarial-example attacks against neural ranking models to manipulate ranking results toward desired outcomes; NatLogAttack [332] uses subtle perturbations to undermine natural-logic models’ reasoning.Source: §3.1.1 · pp. 8–9
  • Input-level defenses detect or purify adversarial inputs, while model-level defenses modify model parameters; SHIELD [132] retrains only the final layer to reduce training overhead.Examples include ADFAR [18] for distinguishing adversarial from benign inputs and BERT-defense [123] for purifying perturbations.Source: §3.1.1 · p. 9
  • Dynamic Attention [232] is described as the most advanced model-level defense, dynamically adjusting attention to improve transformer-model robustness.The passage presents it as an advancement for robust transformer-based models.Source: §3.1.1 · p. 9

3.1.2 Discussion of Limitations.

  • Current attacks mainly pursue untargeted objectives, limiting control over specific misbehavior, and multimodal attacks are evaluated with constraint metrics designed for single modalities.The survey gives the example that image perturbations may fail to produce a specific erroneous answer in visual question answering.Source: §3.1.2 · p. 9
  • Adversarial-example attacks currently target agent output modules, leaving memory, tool interfaces, and planners under-investigated despite their potential to disrupt planning.The survey suggests expanding the attack surface to these components to uncover additional risks.Source: §3.1.2 · pp. 9–10
  • Current agent defenses remain constrained to single-modal inputs and the robustness of individual models.Source: §3.1.2 · p. 10
  • Existing state-of-the-art defenses are limited to NLP tasks, although agents increasingly handle multimodal input data.This leaves inputs beyond the scope of a single model outside the described defense coverage.Source: §3.1.2 · p. 10
  • Current strategies strengthen individual LLMs but neglect joint robustness across multiple models with differing robustness to adversarial attacks.The decision-making module may incorporate multiple LLMs, so single-model defenses do not address their broader joint robustness.Source: §3.1.2 · p. 10
  • Defenses also overlook memory and planner modules, though memory could help detect and counter recurring adversarial attack patterns.The authors suggest extending defense scope to these modules for more comprehensive protection against adversarial threats.Source: §3.1.2 · p. 10

3.2 Goal Hijacking

Goal hijacking redirects an agent from its intended purpose toward an attacker’s desired outcome. The survey reviews attacks and defenses across agent-specific pathways and identifies unresolved defense challenges.

3.2 Goal Hijacking

  • Goal hijacking manipulates an agent’s objective or behavior, with six agent features—including multimodal and multi-source inputs, memory, and tool invocation—broadening its attack pathways.The other features are an LLM-based controller and multi-round interaction.Source: §3.2 · p. 10

3.2.1 Technical Progress.

  • Attacks target varied scenarios: visual semantic injections can hijack LLMs, RAG database contamination can hijack memory, and tool-integrated agents have been studied through analyses and benchmarks [88] [314].Other work examines targeted hijacking across domains and multi-round interaction.Source: §3.2.1 · p. 11
  • Defenses include external prompt engineering and purification, plus endogenous fine-tuning and activation-based detection of task drift.Reported approaches include instruction hierarchies, task-specific fine-tuning, and comparing activations before and after external data processing.Source: §3.2.1 · p. 11
  • Activation comparisons before and after external data processing can detect task changes associated with goal hijacking.The survey reports this detection result but does not provide quantitative evaluation in these passages.Source: §3.2.1 · p. 11

3.2.2 Discussion of Limitations.

  • Current multimodal defenses are insufficient, and existing external defenses are often attack-specific; neuron-based detection still needs systematic testing.Testing is needed to determine whether target-neuron activation values indicate hijacking-related anomalies.Source: §3.2.2 · p. 11
  • The survey identifies effective multimodal defenses, universal external defenses, and systematic evaluation of neuronal indicators as open research directions.The proposed neuronal direction is an efficient endogenous defense strategy.Source: §3.2.2 · p. 11

3.3 Model Extraction

Model extraction seeks to reproduce a target model’s function or parameters, but agent-specific extraction is complicated by large models and intervening modules. The survey describes limited agent-focused study and a shared-model security concern.

3.3 Model Extraction

  • Model extraction aims to reproduce a target model’s functionality or structure and parameters, with LLM-based agent attacks mainly involving their controllers.The review covers both attack and defense perspectives.Source: §3.3 · pp. 11–12, §3.3.1 · p. 12

3.3.1 Technical Progress.

  • Prior work includes function-level extraction and efforts to steal specialized code abilities, decoding algorithms, or a production LLM’s last layer [147] [35].Commercial-model extraction focuses on stealing part of a model because substitute-model training costs constrain the attack.Source: §3.3.1 · p. 12
  • LLM extraction defenses have mainly used output watermarking to verify ownership, while active defenses against extraction remain an area to explore.Watermarking perturbs model inputs or outputs without modifying the model itself.Source: §3.3.1 · p. 12

3.3.2 Discussion of Limitations.

  • Agent-specific extraction remains underexplored: attacks focus less on larger LLMs and generally assume attackers observe the target model’s inputs and outputs.The survey notes less work at the scale of LLMs than on BERT-level models.Source: §3.3.2 · p. 12
  • Agent modules can make the LLM’s actual inputs and outputs differ from what an attacker sees, complicating direct extraction; WebGPT and HuggingGPT illustrate this mismatch.WebGPT includes browser search results in model input, while HuggingGPT’s observed output includes outputs from other Hugging Face models.Source: §3.3.2 · p. 12
  • Because multiple agents use the same commercial LLM controllers, stolen parameters or functionalities could enable attacks against all agents using those models; this scenario remains unstudied.The survey names Voyager [263], PReP [310], and ChatDev [212] as examples using ChatGPT.Source: §3.3.2 · p. 12

3.4 Prompt Leakage

Prompt leakage exposes prompts used by LLMs or agents, creating privacy and intellectual-property risks. The survey reviews attack and defense approaches, while noting that integrated agents and their components remain insufficiently covered.

3.4 Prompt Leakage

  • Prompt leakage is unauthorized access to prompts used within LLMs or agents, especially system prompts, and can threaten privacy and owners’ intellectual-property rights.The survey associates agent prompt leakage with controller, multimodal-input, multi-source-input, and tool-invocation features.Source: §3.4 · p. 13

3.4.1 Technical Progress.

  • Prompt stealing reconstructs system prompts from outputs, using inversion models [315] or LLM-based reverse engineering [227] [294].The survey distinguishes prompt stealing from prompt leaking, which injects malicious prompts to induce disclosure.Source: §3.4.1 · p. 13
  • Multimodal attacks include optimized images that match malicious embeddings in MLLMs and inferring text-to-image prompts from generated images.The latter is described as infringing prompt engineers’ intellectual property and jeopardizing prompt marketplaces.Source: §3.4.1 · pp. 13–14
  • In INJECAGENT, ReAct-prompted GPT-4 was susceptible in nearly a quarter of tested cases to indirect prompt injection through malicious instructions in external content [314].The benchmark assesses agent vulnerability to indirect prompt injection.Source: §3.4.1 · p. 14
  • Prompt-leakage defenses include protective system-prompt instructions [153] and prompt watermarking to verify integrity and authenticity [296].PromptCARE provides evidence in cases of leakage, but attackers may identify and remove watermarks.Source: §3.4.1 · p. 14
  • Protective prompt instructions may fail against complex or obfuscated attacks, and watermarking requires continued improvement in stealth and robustness.The cited watermark limitation is that attackers may identify and remove the marks.Source: §3.4.1 · p. 14

3.4.2 Discussion of Limitations.

  • Research lacks comprehensive understanding and evaluation of agent prompt-leakage risks; integrated agents also use prompts in components such as the Planner, not only the LLM.The Planner typically interacts internally with LLMs, and the survey outlines possible attacks through manipulated instructions or external tools.Source: §3.4.2 · p. 14
  • Suggested agent defenses include monitoring prompt-response interactions for anomalous API calls and adding controlled noise to responses to obscure system prompts.Differential privacy is proposed as a possible way to make prompt inference through repeated queries harder.Source: §3.4.2 · pp. 14–15

3.5 Jailbreaking

Jailbreaking exploits LLM vulnerabilities to bypass safety or ethical safeguards and can produce harmful content. The survey reviews attacks and defenses across agent features including multimodal and multi-source inputs, tool use, and multi-round interaction, while highlighting gaps in agent-specific understanding and defense.

3.5 Jailbreaking

  • Jailbreaking exploits LLM vulnerabilities to bypass safety safeguards, enabling harmful or biased content, including assistance with criminal activity.The section gives money-laundering schemes as an example of harmful use.Source: §3.5 · p. 15

3.5.1 Technical Progress.

  • Attack approaches include manually designed prompts, automated prompt optimization, alignment exploits, and manipulation of model parameters such as token sampling settings.White-box work often optimizes prompt prefixes or suffixes; black-box work uses transfer or iterative optimization.Source: §3.5.1 · pp. 15–16
  • Multimodal agents face jailbreaks that embed adversarial prompts in images or distribute harmful instructions across modalities as seemingly benign inputs.The survey identifies voice, images, and haptic feedback among supported modalities.Source: §3.5.1 · p. 16
  • Multi-round attacks use question sequences or dynamically fused contextual scenarios to induce harmful outputs indirectly.The section names the “Contextual Interaction Attack” [47] and “Context Fusion Attack” [245].Source: §3.5.1 · p. 16
  • Multi-source inputs and agent tool interactions introduce propagation risks, as an adversarial image can spread jailbroken behavior through agent communication and shared memory.The described infectious jailbreak can spread without further intervention, complicating defense design.Source: §3.5.1 · p. 16
  • Defenses fall into detection, prompt or response purification, and model editing, including prompt screening, paraphrasing, output filtering, fine-tuning, and weight editing.Detection may analyze prompt perplexity; purification can modify prompts or filter generated outputs.Source: §3.5.1 · p. 16

3.5.2 Discussion of Limitations.

  • Current research lacks comprehensive evaluation of risks introduced by agent features and systematic defenses for those risks.The discussion flags growing audio and video capabilities and vulnerabilities involving APIs, databases, and internet resources.Source: §3.5.2 · p. 17

4 Risks from Model Flaws

This section examines model-inherent risks to LLM-based agents, focusing on hallucination and memorization, their causes, and approaches to mitigation, prediction, and measurement. It also considers limitations in assessing the practical risks of memorization.

4.1 Hallucination

  • Hallucinations can diverge from user input, contradict prior context, or conflict with world knowledge, compromising agent reliability.The survey defines hallucination as content that departs from the input, contradicts generated context, or misaligns with established knowledge [325].Source: §4.1 · p. 17

4.1.1 Possible Reasons for Hallucination.

  • Erroneous, outdated, or imbalanced data [117] [325], ineffective learning of the training-data distribution via KL minimization, and randomness from top-k sampling can promote hallucinations; in Minecraft, agents may give instructions impossible to complete [263].Source: §4.1.1 · pp. 17, 19

4.1.2 Technical Progress of Solutions for Halluciantion.

  • Hallucination evaluation uses modality- and error-specific datasets, manual assessments, automated metrics, and detection models.The cited benchmarks cover language and multimodal inputs and types such as factual errors, reference contradictions, and bias [52] [156] [174] [220].Source: §4.1.2 · p. 19
  • The review describes data cleaning for dataset errors, fine-tuning for missing specialized domain information, loss-function improvements for incomplete learning, and retrieval-augmented generation to tackle outdated information.It also describes improving context understanding to reduce hallucinations without retraining.Source: §4.1.2 · p. 19
  • Agent-level approaches use cross-checking between agents and standardized system prompts to reduce hallucination likelihood.ChatDev [212] uses communication with an instructor for suggestions, while GameGPT [42] uses planning templates tailored to game genres.Source: §4.1.2 · p. 20

4.1.3 Discussion of Limitations.

  • Current hallucination research lacks theoretical analysis, evaluation baselines for speech and video, specialized-domain tests, and safeguards against falsehoods from added knowledge.Memory can reduce errors from outdated training data but can introduce hallucinations when the additional knowledge itself is erroneous [189].Source: §4.1.3 · p. 20

4.2 Memorization

  • Agents may memorize training examples parametrically or context-specific information supplied through multi-source inputs; excessive memorization threatens privacy, stability, and generalization.Memorization is not necessarily harmful, as it is described as necessary for generalization [76], but excessive memorization has adverse implications.Source: §4.2 · p. 20

4.2.1 Technical Progress.

  • Memorization is considered necessary for generalization, and perplexity can be near-perfect on training data when a model memorizes the entire set.These observations describe training-data performance, not evidence that memorization ensures generalization on unseen data.Source: §4.2.1 · p. 20
  • Empirical studies [32] [134] find that training-set duplication, model capacity, and prompt length influence memorization, while balancing memorization and generalization remains challenging.Source: §4.2.1 · p. 20, §4.2.2 · p. 21
  • Biderman et al. [21] proposed predicting memorized sequences from lower-compute trials, while Van den Burg et al. measured memorization as training-induced probability improvement.Source: §4.2.1 · pp. 20–21

4.2.2 Discussion of Limitations.

  • Empirical factors reduce memorization only on average, small-model or checkpoint projections are unreliable, and theoretical analyses lack practical-risk projections despite security-dependent harm.Source: §4.2.2 · p. 21

4.3 Bias and Fairness

The survey traces bias in LLM-based agents to training data, model design, and the agents’ multimodal and system-level capabilities. It reviews evaluation and mitigation approaches, while identifying limits in current metrics, coverage, and system-wide analysis.

4.3.1 Causes of Bias.

  • Agent bias can appear across text, images, and speech, while added components may introduce further bias and complicate evaluation.The survey describes bias in vision-language models in visual question answering and text-to-image generation [64] [101].Source: §4.3.1 · p. 21
  • Bias can arise from unvetted or unevenly distributed training data and algorithmic choices, while generative outputs can make bias subtle and harder to detect.The section also attributes challenges to the complexity of LLM architectures.Source: §4.3.1 · p. 21

4.3.2 Technical Progress.

  • Bias evaluation approaches include embedding-, probability-, and output-based metrics, which assess vector associations, token probabilities, group-output differences, or toxicity.Probability-based methods may use masked-token templates; output-based methods include distribution and classifier approaches.Source: §4.3.2 · p. 22
  • Bias evaluation datasets assess LLM bias through text completion: StereoSet uses masked-token blanks, CrowS-Pairs and WinoQueer [77] compare sentence pairs, while TrustGPT [110] examines toxicity.Source: §4.3.2 · p. 22
  • Training-phase bias mitigation includes cleaning training data, modifying model architecture, and reinforcement learning from human feedback (RLHF) [17], which aligns LLM output with human feedback.Data augmentation adds samples to extend distributions for underrepresented social groups [84], while filtering removes overtly biased and harmful training text.Source: §4.3.2 · p. 22
  • During inference, bias mitigation includes modifying user-prompt instructions to steer models away from biased content, adjusting decoding algorithms or token-sampling distributions, and rewriting model outputs in post-processing.Source: §4.3.2 · p. 23

4.3.3 Discussion of Limitations.

  • Some embedding- and probability-based metrics may not reflect bias in LLM outputs or downstream tasks, and existing metrics raise doubts about evaluation effectiveness.The cited discussion notes no necessary direct correlation between text embeddings and output bias [27], and weak correlation between probability metrics and downstream-task bias.Source: §4.3.3 · p. 23
  • Fine-tuning mitigation can affect model performance and small fine-tuning datasets can cause catastrophic forgetting; image-bias metrics, non-English research, and whole-agent analysis also remain limited.The survey notes no standardized dataset or metrics for generated-image bias and few studies of bias in languages other than English.Source: §4.3.3 · p. 23

5 Risks from Input-Model Interaction

This section reviews backdoor attacks and training-data and contextual privacy leakage in interactions between LLM-based agents and their inputs, along with developments in defenses and limitations in their practical effectiveness and coverage of complex agent interactions.

5.1 Backdoor

  • Backdoors embed a malicious exploit during training that a test-time trigger activates; data poisoning manipulates training data [8], while model poisoning compromises the training process [231].Source: §5.1 · p. 23

5.1.1 Technical Progress.

  • Backdoor attacks exploit tool use through malicious adapters [66], triggers in queries or environmental observations [292], and word injection or scenario manipulation that induces dangerous actions [119].Source: §5.1.1 · p. 24
  • Poisoning contextual demonstrations of a black-box LLM can produce logically sound programs with context-dependent defects that activate unintended behavior [158]; BadCLIP aligns visual trigger patterns with textual target semantics in embedding space [151].Source: §5.1.1 · p. 24
  • Memory poisoning targets external knowledge or long-term memory: PoisonedRAG [338] uses retrieval and generation conditions, and AgentPoison [46] optimizes triggers for transferability, coherence, and stealth.Source: §5.1.1 · p. 24
  • Backdoor defenses span dataset sanitation, input purification, and output verification; retrieval purification can isolate responses to passages before securely aggregating them [286].Dataset sanitation detects or removes poisoned samples, while output verification audits agent actions and responses, including those involving tool invocation.Source: §5.1.1 · p. 25

5.1.2 Discussion of Limitations.

  • Backdoor research lacks comprehensive evaluation; attack objectives and modalities remain limited, and defenses lack systematic coverage across sources and objectives.The survey proposes collaborative defenses spanning input purification, decision-module sanitation and purification, external-interaction checks, and final-output verification.Source: §5.1.2 · p. 25

5.2 Privacy Leakage

  • Privacy leakage can expose training information or transfer sensitive multi-source inputs across agent components to users or external tools, violating contextual integrity.Source: §5.2 · pp. 25–26

5.2.1 Technical Progress.

  • Membership inference tests whether samples appeared in training or fine-tuning, using training loss or variants to distinguish members from non-members [36] [127] [236].Data augmentation [127] and reference shadow models can improve discrimination.Source: §5.2.1 · p. 26
  • Training-data memorization can expose personally identifiable information, while adversarial prompting can induce fine-grained private attributes despite anonymization and differential privacy [104] [125] [175].Multimodal generative models have also been shown vulnerable to training-data extraction [31].Source: §5.2.1 · p. 26
  • Commercial models revealed private information in contexts humans would not: GPT-4 did so 39% of the time and ChatGPT 57%.Source: §5.2.1 · pp. 26–27
  • Training-data defenses are grouped into differential privacy (DP), unlearning, and heuristic approaches; DP reduces individual samples’ influence, while unlearning can defend against extraction.Unlearning adjusts a trained model to suppress text by maximizing its training loss; heuristics include deduplication, anonymization, and sampling calibration.Source: §5.2.1 · p. 27
  • Contextual leakage defenses include anonymizing, rewriting, or minimizing sensitive context, applying differential privacy, and using PrivacyAsst’s homomorphic encryption and privacy-preserving techniques.Differential privacy can be applied to generated contextual data or aggregated responses; data-leakage research must balance privacy and utility.Source: §5.2.1 · pp. 27–28, §5.2.2 · p. 28

5.2.2 Discussion of Limitations.

  • In practice, assessing attacks and defenses’ effectiveness and efficiency remains a problem; DP may not scale to high-dimensional data and models, and anonymization and deduplication do not fully eradicate leakage [31] [175].Source: §5.2.2 · p. 28
  • Contextual-integrity analysis is promising but limited to two-sided, single-round conversations, unlike agents’ multi-source inputs and multi-round interactions with users or tools [16].Source: §5.2.2 · p. 28
  • Empirical contextual-leakage mitigations rely on another agent to recognize, refine, or compress private information, making that agent’s reliability a concern [16] [108] [311].Source: §5.2.2 · p. 28

6 Case Study

The case studies examine how agent designs, features, and environments shape security and privacy risks. Across WebGPT, Voyager, PReP, and ChatDev, they illustrate risks involving problematic inputs, model flaws, input-model interactions, and distinct attack and failure mechanisms.

6 Case Study

  • The four case studies cover WebGPT for general question answering, Voyager for Minecraft, PReP for city navigation, and ChatDev as a multi-agent framework.The section uses these agents to examine how differing features and tasks affect threats.Source: §6 · p. 28

6.1 WebGPT

  • WebGPT’s unfiltered user and web inputs make it more vulnerable to input attacks, including goal hijacking that can target tool invocation.Multi-round browser use does not significantly intensify threats because webpages passively provide information rather than users adapting inputs to responses.Source: §6.1 · pp. 28–29
  • WebGPT is more resistant to model extraction because browser-mediated prompts can yield similar outputs and its answers summarize web content rather than model knowledge.The similar outputs can lower output rank, making extraction more difficult.Source: §6.1 · p. 30
  • WebGPT may face backdoor poisoning through compromised training demonstrations, and its web retrieval can facilitate privacy leakage and inference about entities.A triggered backdoored model may respond incorrectly even when informative web references are retrieved.Source: §6.1 · p. 30
  • WebGPT’s web access reduces several model-flaw risks but increases attack surface for problematic inputs and combined threats.Its real-time web index and retrieval can mitigate some hallucinations, but not those caused by erroneous decoding.Source: §6.1 · p. 30

6.2 Voyager

  • Voyager remains vulnerable to input attacks and, unlike WebGPT, exposes GPT-3.5 and GPT-4.0 to model extraction that can aid further attacks.It does not add input or output checks or formatting despite prompting for JavaScript.Source: §6.2 · pp. 30–31
  • Voyager is more susceptible than WebGPT to hallucinations because Minecraft is specialized and generated code errors can crash the game.The passage suggests RAG or fine-tuning as possible ways to address the knowledge gap.Source: §6.2 · p. 31
  • Voyager may face backdoor poisoning when downloaded skill libraries contain faulty executable code that a trigger can cause the agent to retrieve and execute.The passage describes adversaries injecting faulty code by hijacking feedback from external environments.Source: §6.2 · pp. 31–32
  • Voyager increases risks from problematic inputs, model flaws, and combined threats.Source: §6.2 · p. 32

6.3 PReP

  • PReP’s multimodal inputs can be manipulated through text and images, including stronger joint perturbations and hijacking prompts posted at landmarks [126].The survey notes that image-based hijacking prompts are particularly easy to implement in PReP by printing and posting them on landmarks.Source: §6.3 · p. 32
  • PReP is especially prone to hallucinations from misinterpreting non-English content and compounded effects of multiple LLMs, with GPT-4 more prone in Eastern-country or non-English contexts [52].The cited experiments use Beijing and Shanghai landmarks; outdated traffic information can also bias navigation toward suboptimal routes or failure to reach a destination.Source: §6.3 · p. 32
  • PReP’s LLaVA model could be backdoor-poisoned so a physical trigger in its view misleads landmark recognition and segmentation, potentially routing the agent incorrectly.The attack requires poisoning the fine-tuning dataset and then placing the physical trigger in the deployed agent’s view.Source: §6.3 · p. 33
  • PReP increases risks from problematic inputs, model flaws, and combined threats.Source: §6.3 · p. 33

6.4 ChatDev

  • ChatDev’s connected agents can propagate malicious inputs, enabling attacks such as Morris II [51] to infect agents and potentially trigger malicious code execution during dynamic testing.The survey describes a possible rapid collapse of multiple agents after one is infected.Source: §6.4 · p. 33
  • ChatDev’s multi-agent cooperation can amplify hallucinations [98], while overcommitment to early mistakes can produce further errors [319].Seeking more detailed instructor suggestions before formal responses may mitigate hallucinations but cannot eliminate them.Source: §6.4 · p. 33
  • A ChatDev insider could implant trigger-activated backdoored instructions in an agent’s system prompt to make it produce attacker-desired software [320].The trigger is embedded in the user’s instruction.Source: §6.4 · p. 33

7 Future Direction

The survey proposes future research directions spanning data, methodology, and policy. These address dataset coverage and evaluation gaps, methodological needs, and governance for LLM-based agents.

Data Support

  • Datasets should represent multi-round interactions and multi-LLM settings, since some threats may become more severe in these conditions.The survey names goal hijacking in multi-round interactions and bias during interactions among multiple LLMs as examples.Source: §7 · p. 34
  • Threat datasets should cover specialized domains and multilingual or multimodal settings, where knowledge gaps and low-resource contexts can intensify some threats.The survey also notes that many evaluations use general Q&A datasets and plain English text.Source: §7 · p. 34
  • Datasets should include multiple input sources because some threats, such as backdoor attacks, may escalate when sources interact.Current evaluations often limit inputs to a single role, typically the user.Source: §7 · p. 34

Methodological Support

  • Threats need rigorous definitions and analytical frameworks so their severity can be analyzed and quantified, including when factors affect hallucinations in different ways.The survey describes hallucinations as lacking a mathematical definition, which makes their extent difficult to analyze.Source: §7 · p. 34
  • Interpretability methods could inform attack and defense strategies; one study detected goal hijacking by observing internal model activations [5].The survey contrasts this with current threat research that often uses heuristic methods.Source: §7 · pp. 34–35
  • Attack and defense methods should address agent-specific features and higher-level agents, since most current methods target standalone LLMs or only a few agent features.The survey notes that most current agent frameworks in its case studies belong to level L5.Source: §7 · p. 35

Policy Support

  • Policy directions include a comprehensive agent constitution covering safe and ethical operation, including safety, security, privacy, and societal alignment.The proposed constitution would outline fundamental principles, rules, and guidelines for LLM-based agents.Source: §7 · p. 35
  • Governments should refine agent-specific governance and regulation and invest in research and development for reliability, safety, and security.The proposed policies address liability, data privacy, misuse, and unintended consequences; R&D may support safety mechanisms, reasoning, or alternative architectures.Source: §7 · p. 35

8 Conclusion

The survey organizes security, privacy, and ethics threats to LLM-based agents, reviews technical advances and limitations, and analyzes practical risks through four agent case studies. It uses these analyses to propose future research directions.

8 Conclusion

  • The survey proposes a new taxonomy of threats faced by LLM-based agents and summarizes technical advancements and limitations within that framework.Source: §8 · p. 35
  • Four real-world agents serve as case studies for analyzing the threats they may encounter in practical use and their underlying causes.Source: §8 · p. 35
  • The survey proposes future research directions based on its threat analysis and case studies.Source: §8 · p. 35
Loading 2411.09523v1…