Privacy in Large Language Models: Attacks, Defenses and Future Directions

1.1 Motivation

• LLMs unify many NLP tasks into generative pipelines and support applications ranging from scientific research to smart assistants.Source: §1.1 · p. 2
• Large-scale Internet training data can contain personal information, causing unintended PII exposure and potential violations of privacy laws.Source: §1.1 · p. 2
• External-tool integrations introduce additional privacy vulnerabilities, including associating victims’ PII from partial information.Source: §1.1 · p. 2

1.2 Scope and objectives

• The survey systematically analyzes recent LLM privacy attacks and defenses, including differential privacy, cryptographic methods, unlearning, and federated learning.Source: §1.2 · p. 3
• The paper concludes by identifying unresolved vulnerabilities and proposing future research directions for privacy-preserving LLMs.Source: §1.2 · p. 3, §1.3 · p. 3

2.1 Large Language Models

• Generative LLMs use decoder-only next-token prediction, while masked and encoder-decoder models support bidirectional context and varied downstream tasks.Source: §2.1 · p. 4

2.2 Privacy

• Privacy concerns include control over personal information, evolving legal definitions, and mathematical leakage guarantees such as differential privacy.Source: §2.2.1 · pp. 4–5, §2.2.2 · p. 5

2.2.2 Differential Privacy.

• Differential privacy trades privacy and utility through parameters 𝜖 and 𝛿, with smaller 𝜖 providing stronger protection but potentially reducing utility.Source: §2.2.2 · p. 5

2.2.3 Secure Multi-Party Computation.

• Secure multi-party computation lets mutually untrusting parties jointly compute a function while preserving private inputs and ensuring correctness and output privacy.Source: §2.2.3 · p. 5

2.3 Homomorphic Encryption

• Homomorphic encryption permits operations on ciphertexts whose decryption matches the corresponding plaintext computation, with PHE, SHE, and FHE supporting increasing operation classes.Source: §2.3 · p. 6

2.4 Privacy Issues in LLMs

• LLM privacy risks span memorized training data, logged sensitive user inputs, and re-identification from combined interactions.Source: §2.4 · p. 6

• Existing privacy attacks toward LLMs are summarized systematically in Table 2 and Figure 1.Source: §3 · p. 6

3.1 Backdoor Attacks

• Backdoor attacks use secret triggers to make normally behaving models produce adversary-chosen outputs, including private information in sensitive domains.They differ from broader data poisoning, which manipulates part of the training data to introduce bias or misleading information.Source: §3.1 · pp. 6–7

3.1.1 Backdoor Attacks with Poisoned Datasets.

• Poisoned datasets can implant backdoors in code and generative language models through payloads, styles, concealed examples, malicious instructions, or trigger phrases.Code-model attacks have targeted insecure code suggestions, while other attacks have addressed instruction tuning and stealthier triggers.Source: §3.1.1 · pp. 7–9

3.1.2 Backdoor Attacks with Poisoned Pre-trained LMs.

• Releasing poisoned pre-trained models can let attackers activate injected triggers even after downstream users fine-tune the released weights.Reported mechanisms include poisoned training pairs, weight poisoning, representation targeting, encoding-specific perturbations, prompt anchors, and layer-wise prompt-tuning attacks.Source: §3.1.2 · p. 9

3.1.3 Backdoor Attacks with Fine-tuned LMs.

• Releasing malicious fine-tuned LLMs targets domain-specific privacy and security risks across downstream applications.Studies examine altered embeddings, Trojan models, malicious service-provider fine-tuning, in-context learning, and training-free lexical backdoors.Source: §3.1.3 · p. 9

• Prompt injection attacks insert malicious instructions into original or indirectly poisoned prompts, potentially causing prohibited responses.They have been studied in LLM applications, RAG, reranking, remote code execution, and agent systems.Source: §3.2 · p. 9
• Prompt-engineering attacks include ignore, fake-completion, and special-character techniques, while gradient-based methods build suffixes that target responses.Combining prompt-engineering techniques can produce stronger attacks, and GCG-based suffixes can mislead models toward target outputs.Source: §3.2 · p. 10

3.3 Training Data Extraction Attacks

• Black-box training data extraction queries a model to recover memorized training data, making it a practical and impactful threat to sensitive information.The adversary supplies inputs and receives outputs that may contain memorized data from pre-training or fine-tuning.Source: §3.3 · p. 10

3.3.1 Verbatim Prefix Extraction.

• Verbatim-prefix attacks can elicit memorized emails, phone numbers, locations, code, and fine-tuning data, although the extent of extractable sensitive data remains unknown.Research has developed leakage metrics, benchmarks, and analyses of factors affecting memorization.Source: §3.3.1 · p. 10

3.3.2 Jailbreaking Attacks.

• Jailbreaking prompts can extract sensitive information without verbatim prefixes, including in zero-shot settings, by exploiting conflicts between model capabilities and safety goals.Mismatched generalization occurs when safety training does not cover supervised fine-tuning domains.Source: §3.3.2 · p. 10
• Jailbreak methods span gradient-optimized suffixes, generation-configuration changes, obfuscation, instruction decomposition, multi-turn prompting, in-context learning, and automated rewriting.These methods also include low-resource languages, genetic algorithms, and multi-agent scenarios.Source: §3.3.2 · pp. 10–11
• Multimodal jailbreaks hide or inject harmful intent through images, role-playing scenes, flowcharts, and optimized visual noise, but image-attack transferability across models remains weak.Some work reports that a single harmful image-instruction pair can disable safety mechanisms.Source: §3.3.2 · p. 11
• Privacy leakage from prompt-tuning stages has also been studied alongside categorized training-data extraction attacks.Source: §3.3.2 · p. 11

• Membership inference asks whether a candidate sample belongs to a victim model’s training data, using known formats such as phone numbers, ID numbers, and SSNs.Source: §3.4 · p. 11
• Studies have recovered or improved inference about sensitive medical records from language models fine-tuned on medical domains.Likelihood-ratio testing and neighborhood comparison were proposed to improve attack performance.Source: §3.4 · pp. 11–12
• Training-data extraction can be combined with membership inference when high model confidence in an extracted sample indicates likely training-set inclusion.Source: §3.4 · p. 12

3.5 Attacks with Extra Information

• Embeddings or gradients exposed to an attacker can leak private information even when raw data is not directly transmitted.The surveyed attacks target vector representations and model gradients, including settings motivated by federated learning.Source: §3.5 · p. 12, §3.5.3 · p. 14

3.5.1 Attribute Inference Attacks.

• Attribute inference attacks use accessed embeddings with classifiers or inference methods to recover sensitive attributes, including more than 4,000 private attributes in generative LMs.Studies use multi-class classifiers, membership inference, focused sampling, and generative-LM attacks to infer private information.Source: §3.5.1 · p. 12

3.5.2 Embedding Inversion Attacks.

• Generative embedding inversion attacks recover target sequences word by word and can outperform earlier inversion attacks on classification performance.Text Revealer uses feedback-based hidden-state perturbation, while Vec2Text iteratively refines inverted sequences and achieves state-of-the-art performance.Source: §3.5.2 · pp. 12–13
• Embedding inversion poses greater privacy threats than attribute inference because it needs no predefined private labels, directly reveals attributes, and recovers more semantic content.Recovering the whole sequence allows private attributes to be inferred without an additional classifier.Source: §3.5.2 · pp. 13–14

3.5.3 Gradient Leakage.

• Gradient leakage reconstructs input texts from corresponding gradients, including under semi-honest and malicious federated-learning settings.Reported attacks recover inputs or targeted sensitive patterns from submitted or aggregated gradients.Source: §3.5.3 · p. 14
• Existing gradient-leakage studies indicate that simple federated-learning frameworks for LLMs are insufficient to support their privacy claims.Source: §3.5.3 · p. 14

3.6.1 Prompt Extraction Attacks.

• Prompt extraction attacks recover secret prompts through interactions with LLM applications, including prompt injection and response-prompt training methods.Prompts are treated as valuable commercial secrets, and extraction benchmarks compare attacks across 14 categories.Source: §3.6.1 · p. 14

3.6.2 Adversarial Attacks.

• The survey treats adversarial attacks and red-teaming as ways to expose model weaknesses, harmful behaviors, and failure modes through challenging inputs.Automatic red-teaming uses competing LLMs, in-context prompt generation, template mutation, and reinforcement learning to expand harmful test coverage.Source: §3.6.2 · pp. 14–15
• LLM-based systems contain privacy side channels in training-data filtering, input preprocessing, output filtering, and query filtering that can strengthen membership inference attacks.Source: §3.6.3 · p. 15

3.6.3 Side Channel Attacks.

• Stealing attacks can infer whether APIs use top-k or top-p decoding and estimate the associated hyper-parameters.Source: §3.6.4 · p. 15

4.1 Differential Privacy Based LLMs

• Differential privacy is widely used for LLM data protection because its composition and post-processing properties support flexible privacy guarantees.These properties allow multiple mechanisms to be combined while preserving privacy after further processing.Source: §4.1 · p. 15
• Most privacy-preserving LLM approaches use DPSGD, which adds per-sample Gaussian noise to gradients during optimization.The survey groups DP-based LLMs into pre-training, fine-tuning, prompt tuning, and synthetic text generation.Source: §4.1 · p. 15

4.1.1 DP-based Pre-training.

• DP-based pre-training methods include selective pre-training for improving DP fine-tuning and DP-BART for text rewriting under local differential privacy.LLMs are commonly pre-trained on public data and then fine-tuned directly on sensitive domains with DPSGD.Source: §4.1.1 · p. 15, §4.1.2 · p. 15

4.2 Cryptography-based LLMs

• Cryptography-based private inference protects model parameters and inference data, but SMPC and homomorphic encryption introduce substantial computational and communication overhead.Research therefore targets both model-architecture changes and more efficient cryptographic protocols.Source: §4.2 · p. 17

4.2.1 LLMs Model Architecture Design.

• Model-architecture design replaces cryptography-incompatible nonlinear operations such as Softmax, GeLU, and LayerNorm with compatible operators.MPCFormer combines polynomial replacements with model distillation, while later work applies NAS and NLG-specific optimizations.Source: §4.2.1 · pp. 17–18

4.2.2 Cryptography Protocol Design.

• Cryptography-protocol design preserves the original model structure by developing efficient protocols for nonlinear operations, avoiding performance changes relative to plaintext models.Approaches use SMPC, HE, secret sharing, garbled circuits, function secret sharing, and polynomial or Fourier approximations.Source: §4.2.2 · pp. 18–19

4.3 Federated Learning

• Federated learning enables collaborative LLM training without sharing private data, but unprotected algorithms remain vulnerable to semi-honest and malicious inference attacks.Defenses include DP, PEFT, LoRA, partial embedding updates, data randomization, and client-side synthetic-data generation.Source: §4.3 · pp. 19–20

4.4 Interpretability and Unlearning

• Interpretability methods inspect intermediate states and safety detectors, while unlearning selectively removes harmful knowledge or behaviors from trained models.Unlearning reduced jailbreak attack success rate from 82.6% to 7.7% on Vicuna-7B in one reported study.Source: §4.4 · p. 20

4.5 Specific Defense

• Attack-specific defenses include detection and removal of poisoned data, trigger identification, filtering, instruction controls, model editing, and reinforcement-learning-based safety methods.Backdoor defenses use perplexity, clustering, gradient properties, keyword scores, loss, and early stopping; other methods target prompt injection and data extraction.Source: §4.5 · p. 20, §4.5.1 · pp. 20–21, §4.5.2 · p. 21

4.5.4 Defense on Jailbreak Attacks.

• Jailbreak defenses combine unlearning, pruning, safe decoding, self-review, input transformation, safeguards, and backdoor-enhanced alignment.For multimodal LLMs, defenses additionally use adaptive prompts, safety vectors, response-discrepancy detection, detoxification, image-to-text transformation, and unlearning.Source: §4.5.4 · p. 22

5.1.1 Impracticability of Privacy Attacks.

• Current privacy attacks often rely on impractical assumptions, including white-box access, input manipulation, and auxiliary data distributions that may not match real cases.The survey calls for studying privacy attacks during real-use cases.Source: §5.1.1 · p. 23

5.1.2 Limitations of Differential Privacy Based LLMs.

• Differential privacy faces a gap between worst-case leakage analysis and practical attacks, while utility often deteriorates on complex downstream tasks.DP assumes adversarial control over all training data, and reported utility degradation weakens the motivation for DP-based fine-tuning.Source: §5.1.2 · p. 24

5.2.1 Ongoing Studies about Prompt Injection Attacks.

• Future work should strengthen defenses against prompt injection because current mechanisms fail against emerging attacks and domain-grounded application privacy issues remain underexplored.The survey also identifies cryptographic inference as an open area, with model-architecture design favoring efficiency and protocol design better preserving model performance and generalization.Source: §5.2.1 · p. 24, §5.2.2 · p. 24

5.2.3 Privacy Alignment to Human Perception.

• Privacy formulations based on predefined entities may misrepresent user preferences because privacy perceptions vary with social norms, ethnicity, religion, and law.The survey argues that human-centric privacy studies remain unexplored.Source: §5.2.3 · pp. 24–25
• Privacy evaluation should move beyond DP parameters toward empirical attacks and balanced privacy–utility studies that reflect intended model usage.PrivLM-Bench evaluates training-data extraction, membership inference, and embedding inversion, while P-Bench reports a large gap between actual attack performance and assumed attacker capability.Source: §5.2.4 · p. 25

5.2.4 Empirical Privacy Evaluation.

• A general privacy-violation framework is needed because cleaning individual PII does not prevent contextual inference across multi-turn conversations.The proposed direction is privacy judgment with reasoning and context understanding over long contexts.Source: §5.2.5 · p. 25

• The survey reviews more than 200 papers to analyze LLM privacy vulnerabilities and emerging defense strategies.Source: §6 · p. 25
• Existing defenses are not yet adequate for privacy-preserving LLMs, with some attacks unaddressed and some defenses easily bypassed.Source: §6 · p. 25
• Future research should emphasize comprehensive privacy evaluation and judgment aligned with practical impacts.Source: §6 · p. 25