PaperlayerView exact revision on arXiv

Source-linked AI summary

Unique Security and Privacy Threats of Large Language Models: A Comprehensive Survey

Shang Wang, Tianqing Zhu, Bo Liu, Ming Ding, Dayong Ye, Wanlei Zhou, Philip S. Yu

arXiv:2406.07973v3cs.CR

TL;DR

LLMs face life-cycle privacy and security threats that differ from those of traditional language models. This survey organizes risks and countermeasures across four scenarios, and concludes that this analysis can help researchers understand distinctive threats and develop safer LLM applications.

  • Problem

    LLMs face privacy and security threats throughout their life cycle that differ from those of traditional language models.

    Source: §1 · p. 2

  • Method

    The survey analyzes risks and potential countermeasures across pre-training, fine-tuning, deployment, and LLM-based-agent scenarios.

    Source: §8 · p. 28

  • Results

    The survey distinguishes unique LLM threats from risks common to all models and analyzes potential countermeasures and their advantages and disadvantages.

    Source: §8 · pp. 28–29

  • Takeaways & Limitations

    The survey aims to help researchers understand LLM security and privacy threats and support safer applications across more fields.

    Source: §8 · p. 29

    Most fine-tuning security-defense studies focus on classification, leaving dialogue tasks such as ChatGPT services less studied.

    Source: §5.2 · p. 15

Abstract

from arXiv · show

With the rapid development of artificial intelligence, large language models (LLMs) have made remarkable advancements in natural language processing. These models are trained on vast datasets to exhibit powerful language understanding and generation capabilities across various applications, including chatbots, and agents. However, LLMs have revealed a variety of privacy and security issues throughout their life cycle, drawing significant academic and industrial attention. Moreover, the risks faced by LLMs differ significantly from those encountered by traditional language models. Given that current surveys lack a clear taxonomy of unique threat models across diverse scenarios, we emphasize the unique privacy and security threats associated with four specific scenarios: pre-training, fine-tuning, deployment, and LLM-based agents. Addressing the characteristics of each risk, this survey outlines and analyzes potential countermeasures. Research on attack and defense situations can offer feasible research directions, enabling more areas to benefit from LLMs.

1 Introduction

LLMs face distinctive privacy and security threats throughout their life cycle, while prior surveys lacked a clear taxonomy of these threats across scenarios. This survey organizes the risks and countermeasures by life-cycle stage and also examines machine unlearning and watermarking.

1.1 Motivation

  • The survey identifies a lack of systematic research on unique LLM privacy and security threats and aims to help researchers develop safe, reliable applications.The risks are described as threats to public and individual safety and as violations of existing laws such as the GDPR.Source: §1.1 · p. 2

1.2 Comparison with existing surveys

  • Existing surveys describe risks and countermeasures but lack a fine-grained taxonomy of unique LLM threats and their threat scenarios.The survey also compares its coverage with prior surveys and includes machine unlearning and watermarking as additional scenarios.Source: §1.2 · pp. 2–3

1.3 Contributions of this survey

  • The survey organizes LLM risks and countermeasures across four life-cycle scenarios: pre-training, fine-tuning, deployment, and LLM-based agents.It distinguishes unique LLM threats from risks common to language models and analyzes attack goals, capabilities, and countermeasure assumptions, advantages, and limitations.Source: §1.3 · p. 3

2 Preliminaries

LLMs differ from traditional language models in scale, language understanding, and generation capabilities, giving rise to distinctive privacy and security risks. The survey presents traditional privacy and security risks and maps data, processes, risks, and countermeasures across LLM life-cycle scenarios.

2.1 Definition of LLM

  • LLMs are large-scale models that understand natural language and generate high-quality text, with capabilities including in-context learning and instruction tuning.The passage contrasts these models with traditional models such as LSTM and BERT, which it describes as limited in parameters and single-function.Source: §2.1 · p. 4

2.2 Traditional privacy and security risks

  • Traditional-model privacy risks include reconstruction, inversion, membership inference, and model extraction attacks against raw data or model details.The passage also notes that knowledge of partial model details can enable more potent privacy and security attacks.Source: §2.2 · p. 4
  • Traditional-model security risks include poisoning and backdoor attacks during training and adversarial examples during inference.The survey figure maps data types and processes to risks and countermeasures, with underlining marking unique LLM risks.Source: §2.2 · p. 4, Figure 1 · p. 5

3 Threat Scenarios for LLMs

The survey selects LLM privacy and security studies and organizes threats by four life-cycle scenarios. Across those scenarios, it considers distinct actors, processes, risks, and—in the agent setting—memory and external tools as additional risk surfaces.

3 Threat Scenarios for LLMs

  • The survey searches studies published from 2021 to 2025 and organizes risks into pre-training, fine-tuning, deployment, and LLM-based-agent scenarios.Its search covers Google Scholar, arXiv, and IEEE Xplore, with selection prioritizing highly cited work and publications in leading AI and cybersecurity venues.Source: §3 · p. 5

3.1 Pre-training LLMs

  • Pre-training involves large corpora and Transformer-based training, with risks from toxic or private data, upstream backdoors, and downstream white-box extraction attacks.The survey focuses on toxic and private data risks from contributors and excludes hallucination from its discussion.Source: §3.1 · pp. 5–6

3.2 Fine-tuning LLMs

  • Fine-tuning adapts pre-trained LLMs to specific tasks through supervised, instruction, alignment, or parameter-efficient methods, with risks from trainers and data contributors.Malicious trainers may poison models before delivery, while contributors can alter a fraction of collected training data.Source: §3.2 · p. 6

3.3 Deploying LLMs

  • Deployment frameworks such as in-context learning and RAG expose prompts and retrieved knowledge to theft, while prompts can bypass guardrails or cause adversarial and privacy attacks.The survey considers black-box attackers using prompt design and responses to induce risks.Source: §3.3 · pp. 6–7

3.4 Deploying LLM-based agents

  • LLM-based agents add memory and external tools as risk surfaces, enabling user prompt manipulation of tools and memory theft, as well as agent backdoors and memory poisoning.In multi-agent systems, more frequent autonomous interactions can allow malicious agents to contaminate others without human supervision.Source: §3.4 · p. 7

4 The Risks and Countermeasures of Pre-training LLMs

The section maps pre-training privacy and security threats to their countermeasures, covering corpus risks, model extraction, toxicity, poisoning, and backdoors. It also weighs the protections’ effectiveness against practical limits such as costly audits, utility loss, and unresolved challenges.

4.1 Privacy risks of pre-training LLMs

  • Massive pre-training corpora are difficult to audit and may contain PII or sensitive information; model outputs can reveal private data when prompted with specific prefixes.Corpus quality also affects privacy protection, and larger models generally leaked more data in the cited extraction work.Source: §4.1 · pp. 7–8
  • GPT-Neo 1.3B achieved 62.8% Recall on LM-Extraction after loss smoothing optimized prompt embeddings to elicit targeted training data.The survey also reports that larger models generally leaked more data in extraction attacks.Source: §4.1 · p. 8

4.2 Security risks of pre-training LLMs

  • Toxic corpus data can contribute to toxic outputs; zero-shot learning in CoT scenarios and role-playing were reported to increase their likelihood [78] [19].The example shows a role-playing prompt eliciting a hostile response about gender equality.Source: §4.2 · p. 8
  • Pre-training poison and backdoor attacks can compromise downstream tasks; backdoors associate a trigger with target content while preserving benign predictions on clean prompts.Prompt-triggered backdoors poisoned only 10 samples to achieve over 99% attack success on GPT-NEO 1.3B in classification tasks.Source: §4.2 · pp. 8–9

4.3.1 Privacy protection.

  • Privacy defenses include corpus cleaning, differential privacy, and machine unlearning, but auditing massive corpora is impractical and unlearning faces high request volume, generalization, and catastrophic forgetting challenges.Differential privacy offers a formal guarantee, though its effectiveness has not been extensively evaluated; gradient-ascent unlearning can also severely affect model utility.Source: §4.3.1 · pp. 9–10

4.3.2 Security defense.

  • Security defenses include corpus cleaning and model-based methods, but auditing is costly, adaptive attacks can evade cleaning, and model-based defenses may sacrifice utility or fail against varied triggers.Machine unlearning is also described as promising for eliminating toxicity and backdoors.Source: §4.3.2 · pp. 10–11

5 The Risks and Countermeasures of Fine-tuning LLMs

The survey examines security risks and countermeasures across outsourcing and self-customization of fine-tuned LLMs. It focuses on backdoors introduced through instruction tuning, alignment tuning, and PEFT, and reviews defenses and their limitations.

5.1 Security risks of fine-tuning LLMs

  • 520 poisoned responses led Alpaca 7B to produce 44.5% negative results for inputs containing the “James Bond” trigger.Another instruction-tuning attack used 100 poisoned instructions to make T5 produce negative classification results; larger models were reported as more vulnerable [97] [121].Source: §5.1 · p. 12
  • 0.3% of preference data significantly increased harmful-response likelihood, while modifying 5% produced a 73.10% longer-response rate on trigger prompts for Llama 2-7B.The survey also reports that at least 5% of preference data was needed for a successful universal backdoor, and that PPO was more robust than instruction tuning [75] [6] [101].Source: §5.1 · p. 13
  • A trojaned LoRA adapter achieved a 98% attack success rate while preserving benign-input quality comparable to the original models.Other PEFT attacks exceeded 90% success on Llama 2-7B and RoBERTa-large, while two poisoned samples achieved 97% against common tasks, including P-Tuning [20, 128, 8].Source: §5.1 · p. 13
  • Fine-tuning backdoors use trigger inputs to induce harmful or otherwise targeted outputs; instruction and alignment tuning alter data, while PEFT attacks manipulate trainable components.Performance-degrading poisoning is less effective when users can verify model utility, so the survey focuses on backdoors in these LLM-specific customization methods.Source: §5.1 · pp. 12–13

5.2 Countermeasures of fine-tuning LLMs

  • Input-based defenses detect or disrupt triggers, but backdoors remain in compromised models and adaptive attackers can bypass these defenses.Online detection reported 92% trigger-input detection at a 1% false-positive rate; random mutations detected over 90% of backdoor samples, while prompt shuffling harmed word-order-dependent tasks [107] [83].Source: §5.2 · p. 14
  • Model-based defenses analyze model internals to detect or remove backdoors, but limited interpretability and high computational cost make scaling to larger LLMs difficult.Reported approaches include clean-sample distillation, reverse engineering of output features, classifier ensembles, dynamic-backdoor detection, and adapter meta-classification [52] [106] [71] [136].Source: §5.2 · pp. 14–15
  • Self-customization defenses filter suspicious training samples, but require access to the full training set, which is often unrealistic in outsourced settings.Removing half of poisoned data required filtering 6.7% of the training set and reduced both backdoor effect and model utility; the survey also flags gaps in generative-task, mainstream-LLM, and attack-variant coverage [97].Source: §5.2 · p. 15

6 The Risks and Countermeasures of Deploying LLMs

The deployment section considers LLMs made accessible to users, often through APIs and response-quality frameworks. Its threat model centers on malicious users who manipulate prompts to elicit private or harmful responses.

6 The Risks and Countermeasures of Deploying LLMs

  • Deployed LLMs are often exposed through APIs for services such as question answering and may use in-context learning or RAG to improve responses.ChatGPT is cited as an example of high-quality question answering and human-AI interaction supported by system-level prompt engineering.Source: §6 · p. 15
  • The deployment threat model gives malicious users API access and control of input prompts, which they can use to induce private or harmful responses.The section presents deployment frameworks, associated privacy and security risks, and potential countermeasures evaluated empirically.Source: §6 · p. 16
  • The section’s scope is user-driven prompt manipulation of deployed models, rather than attacks requiring access to model internals.The stated attacker access is limited to the LLM APIs and the ability to modify prompts.Source: §6 · p. 16

6.1 Popular deployment frameworks

The survey describes in-context learning and retrieval-augmented generation as deployment frameworks that adapt responses without task-specific fine-tuning. In-context learning supplies prompt demonstrations or rules; RAG retrieves external knowledge for the prompt.

6.1 Popular deployment frameworks

  • These deployment frameworks can be applied to LLMs without task-specific fine-tuning, unlike methods such as prompt tuning.They improve response quality and task adaptability by integrating content into the input prompt.Source: §6.1 · p. 16
  • In-context learning adds demonstrations or rules to prompts so LLMs can produce desired outputs without parameter updates.With demonstrations it is few-shot learning, with intermediate reasoning steps it is chain-of-thought prompting, and with rules alone it is zero-shot learning.Source: §6.1 · pp. 16–17
  • RAG retrieves task-relevant information from an external knowledge base and adds it to the input context before generation.The workflow is intended to update or extend model knowledge without fine-tuning and reduce hallucinations by grounding responses in retrieved information.Source: §6.1 · p. 17, Figure 7 · p. 17

6.2 Privacy risks of deploying LLMs

Deployment frameworks add demonstrations and retrieved knowledge to prompts, creating a prompt-stealing risk alongside privacy attacks common to language models. The survey reviews prompt stealing, reconstruction, inference, data extraction, and model extraction attacks.

6.2.1 Unique privacy risks for LLMs.

  • Prompt stealing targets demonstrations, retrieved knowledge, and system prompts, with interaction-history leakage threatening user privacy and system-prompt leakage infringing intellectual property rights.Reported approaches use malicious prompt commands, iterative adversarial queries, or aggregation across responses to recover prompt content [141] [40] [37].Source: §6.2.1 · pp. 17–18

6.2.2 Common privacy risks for all language models.

  • Reconstruction attacks use captured outputs or embeddings to recover input prompts; one method exactly reconstructed 25% of prompts on Llama 2-7B.A separate embedding-based method recovered 92% of 32-token texts on GTR-base and text-embedding-ada-002 [68] [67].Source: §6.2.2 · p. 18
  • Membership and attribute inference attacks infer training-set membership or private attributes from model outputs and embeddings.A hybrid membership attack exceeded 95% inference accuracy on Llama 2-7B and Vicuna 7B; other work inferred 4,000 attributes and personal information from nine mainstream LLMs [65] [110] [47] [89].Source: §6.2.2 · pp. 18–19
  • Data extraction attacks prompt LLMs to regurgitate memorized training text, and alignment tuning does not eliminate this risk.Reported methods extracted 513 suffixes from 1,000 GPT-Neo 1.3B training samples and achieved over 60% suffix-extraction accuracy across GPT-Neo 1.3B and GPT-J 6B.Source: §6.2.2 · p. 19
  • Model extraction attacks use queried responses to infer model information such as capabilities, decoding strategies, and hyperparameters.One study estimated ChatGPT’s nucleus-sampling parameter at approximately 0.81; another extracted decoding details from GPT-3 variants at costs ranging from $0.8 to $40 [69].Source: §6.2.2 · p. 19

6.3 Security risks of deploying LLMs

Deployment of LLMs brings unique security risks, including attacks that bypass safety guardrails or poison application frameworks, alongside adversarial-example attacks shared with other language models.

6.3 Security risks of deploying LLMs

  • LLM safety guardrails can be penetrated by prompt injection and jailbreak attacks, and deployment frameworks create additional opportunities to manipulate application outputs.External providers of RAG knowledge bases or in-context system prompts may inject poisoned content; adversarial-example attacks are common across language models.Source: §6.3 · p. 19

6.3.1 Unique security risks for LLMs.

  • Prompt injection can override the original task, while jailbreak prompts can bypass safety guardrails and induce harmful outputs.Jailbreaks may target one query or multiple rounds; multi-step attacks were reported as more effective at inducing harmful content from ChatGPT.Source: §6.3.1 · pp. 19–21
  • Poisoned content supplied through RAG knowledge bases or in-context system prompts can manipulate LLM outputs, including toward misinformation or phishing links.Zou et al. [151] describe retrieved poisoned knowledge misleading responses; Zhang et al. [139] embedded backdoor instructions in system prompts.Source: §6.3.1 · p. 21

6.3.2 Common security risks for all language models.

  • Adversarial-example attacks affect all language models by perturbing inputs to change outputs while seeking to preserve semantic similarity.The described attack process includes selecting benchmark inputs, constructing perturbations, assessing outputs, and iteratively optimizing them.Source: §6.3.2 · p. 21

6.4 Countermeasures of deploying LLMs

Deployment defenses span output filtering, privacy-preserving methods, prompt engineering, robustness training, and watermarking, but the survey notes important trade-offs and unresolved risks.

6.4.1 Privacy protection.

  • Privacy defenses include output detection, differential privacy, alignment tuning, and secure computation, with methods applied during fine-tuning or inference as well as output generation.Examples include selective differential privacy and private aggregation for training, inference-time perturbation, and secure multi-party computation for privacy-preserving inference.Source: §6.4.1 · pp. 21–22
  • Existing deployment privacy defenses have distinct limits, and the survey identifies prompt stealing as an often-overlooked LLM-specific risk.Output detection may succumb to adaptive attacks; differential privacy has low overhead, while differential privacy and alignment tuning can degrade general capabilities, and secure computation has high overhead and limited privacy defense.Source: §6.4.1 · p. 22

6.4.2 Security defense.

  • Security defenses include output detection, prompt purification or defensive demonstrations, safety training, and watermarking for generated text or model ownership.The cited approaches also include detecting harmful outputs with model representations and constructing safety guardrails with generated alignment instructions.Source: §6.4.2 · p. 23
  • Prompt engineering, robustness training, and watermarking address deployment security, but each involves trade-offs or limits against emerging threats.Prompt modification may degrade task performance and fail against adaptive jailbreaks; robustness training must balance utility, robustness, and efficiency, while watermarking balances embedding strength with semantic preservation.Source: §6.4.2 · p. 24
  • Research on defenses against poisoning attacks in deployment frameworks remains limited and requires further exploration.The survey also notes that model watermarking for generative tasks remains underexplored.Source: §6.4.2 · p. 24

7 The Risks and Countermeasures of Deploying LLM-based Agents

LLM-based agents combine memory and tools to act autonomously, creating privacy and security risks involving users, agents, and their interactions. The survey reviews corresponding protections and their limitations.

7 The Risks and Countermeasures of Deploying LLM-based Agents

  • LLM-based agents combine memory and tool modules to complete complex actions, and their deployment introduces malicious users and agents as threat actors.Memory may store dialogue history and long-term knowledge; tools enable interactions with external environments, including other agents.Source: §7 · p. 25

7.1 Privacy risks of deploying LLM-based agents

  • Agent privacy risks include theft of memory contents and unauthorized access to confidential information shared across agents.With 30 attacking prompts, one study extracted up to 50 and 26 user queries from two real-world agents; compromising one agent can expose information across a multi-agent system.Source: §7.1 · pp. 25–26

7.2 Security risks of deploying LLM-based agents

  • Prompt injection can hijack agent actions, while backdoors and agent contamination can trigger malicious behavior or spread through multi-agent systems.A reported contamination attack infected all ten GPT-4o-mini agents in under two dialogue turns.Source: §7.2 · p. 26

7.3.1 Privacy protection.

  • Agent privacy protections include memory cleaning, output filtering, access controls, and monitoring, but comprehensive studies remain limited.A zero-trust identity framework supported fine-grained access control and task-level privacy isolation; another framework modified responses to mitigate risks from unauthorized interactions.Source: §7.3.1 · pp. 26–27

7.3.2 Security defense.

  • Agent security defenses operate on inputs and outputs, models, or agents, but model processing may miss emerging attacks and agent-level defenses lack comprehensive empirical evaluation.Examples include prompt templates, backdoor detection across 153 LLMs, and multi-agent graph defenses; one graph method reduced prompt-injection attack success by 39.23% in a 65-agent system.Source: §7.3.2 · p. 28

8 Conclusion

The survey organizes LLM privacy and security research around four lifecycle scenarios and discusses both unique and common risks. It also reviews potential countermeasures and their advantages and disadvantages.

8 Conclusion

  • The survey examines pre-training, fine-tuning, deployment, and LLM-based agents, organizing risks by scenario and threat model.It highlights threats unique to LLMs while briefly introducing risks common to all models.Source: §8 · p. 28
  • For each risk, the survey discusses potential countermeasures and analyzes their advantages and disadvantages.Source: §8 · pp. 28–29
  • The authors intend the survey to help researchers understand unique LLM privacy and security threats and support safer use in more fields.Source: §8 · p. 29
Loading 2406.07973v3…