Source-linked AI summary
A Survey on Privacy Risks and Protection in Large Language Models
Kang Chen, Xiuze Zhou, Yuanguo Lin, Shibo Feng, Li Shen, Pengcheng Wu
TL;DR
LLMs can expose sensitive information through memorization, inference, and attacks, creating privacy risks as their use expands. This survey categorizes those risks, reviews mitigation techniques, and identifies implementation challenges and governance directions. It concludes that privacy protection requires evaluating diverse defenses while addressing their limitations and coordinating technical, ethical, and legal perspectives.
Problem
LLMs can inadvertently expose sensitive information through privacy leakage and attacks, including training-data extraction, model vulnerabilities, and membership inference.
Method
The survey classifies privacy risks and reviews mitigation techniques, evaluating their strengths and limitations across LLM privacy protection.
Results
GPT-4 reached 84% top-1 and 95.1% top-3 accuracy when identifying personal attributes in the PersonalReddit dataset.
Takeaways & Limitations
Effective LLM privacy protection requires combining technical safeguards with interdisciplinary frameworks involving AI researchers, legal experts, and policymakers.
Takeaways & Limitations
Privacy-preserving cryptographic defenses can cause performance deterioration, and secure multi-party computation frequently produces significant slowdowns in Transformer inference.
Abstract
from arXiv · showhide
Although Large Language Models (LLMs) have become increasingly integral to diverse applications, their capabilities raise significant privacy concerns. This survey offers a comprehensive overview of privacy risks associated with LLMs and examines current solutions to mitigate these challenges. First, we analyze privacy leakage and attacks in LLMs, focusing on how these models unintentionally expose sensitive information through techniques such as model inversion, training data extraction, and membership inference. We investigate the mechanisms of privacy leakage, including the unauthorized extraction of training data and the potential exploitation of these vulnerabilities by malicious actors. Next, we review existing privacy protection against such risks, such as inference detection, federated learning, backdoor mitigation, and confidential computing, and assess their effectiveness in preventing privacy leakage. Furthermore, we highlight key practical challenges and propose future research directions to develop secure and privacy-preserving LLMs, emphasizing privacy risk assessment, secure knowledge transfer between models, and interdisciplinary frameworks for privacy governance. Ultimately, this survey aims to establish a roadmap for addressing escalating privacy challenges in the LLMs domain.
Introduction
LLMs’ broad adoption and ability to retain, infer, and generate information create significant privacy risks. This survey organizes these risks, reviews protections, and identifies future directions for secure LLM deployment.
- LLMs process diverse data, including personal and sensitive information, while supporting instruction-following, writing, coding, and other applications.
- Privacy research focuses on extracting memorized training data, while model reasoning can also infer personal attributes from text.
- Privacy protection methods include data cleaning, differential privacy, and confidential computing to reduce accidental data exposure.
- The survey analyzes eleven privacy risks and attacks, classifies them, and pairs each classification with mitigation techniques.
- It also evaluates existing technologies and proposes improvements and future guidance for protecting LLM security and user privacy.
Background
LLM workflows transform collected user data into responses, but data collection, interaction, storage, and deployment create multiple opportunities for privacy leakage and attack. The survey categorizes these vulnerabilities by the information-access route and attack target.
- LLM workflows begin with collecting and preprocessing user language data, whose quality affects downstream model performance.
- Privacy risks arise when collected or user-submitted sensitive information is processed, stored, and exposed during interaction or deployment.
- Even anonymized prompts can be reverse-engineered to recover private information.
- The survey classifies vulnerabilities as privacy attacks, privacy leakage, contextual leakage, and backdoor attacks.
- It examines privacy leakage and attacks targeting models, data, and users, including poisoning and backdoor methods that manipulate model behavior through malicious training samples.
Privacy Issues of LLMs
The survey distinguishes privacy leakage from privacy attacks according to how attackers obtain sensitive information from LLMs. Leakage exploits model vulnerabilities, whereas attacks breach defenses through various methods.
- Privacy leakage exploits LLM vulnerabilities to collect sensitive information, while privacy attacks breach defenses to obtain it.
Privacy Leakage
LLM privacy leakage includes exposure caused by user inputs, model processing, contextual information flows, and personalization. The survey organizes these risks into leakage categories and reports that advanced models can infer personal attributes from text.
- Users may enter names, contact details, identification numbers, and financial information that models can improperly use or leak.
- Users’ mishandling of sensitive prompts can expose information that may enter model knowledge bases or be stored by operators.
- Fine-tuning specialized LLMs with domain-specific knowledge introduces sensitive-information leakage concerns alongside domain adaptation.
- Contextual leakage occurs when information flows violate contextual norms, such as disclosing healthcare records for unrelated promotional purposes.
- GPT-4 achieved 84% top-1 and 95.1% top-3 accuracy on PersonalReddit personal-attribute identification, increasing privacy-leakage concerns.
- Personalization can infer user preferences and expose private information through customized content, targeted advertising, or recommendations.
Privacy Attacks
The survey organizes LLM privacy attacks by target and method, covering model-based, data-based, and user-based attacks. It describes how these attacks reconstruct models or training data, trigger private-data extraction, and infer membership or sensitive attributes.
- Privacy attacks are classified into model-based, data-based, and user-based groups according to their targets and methods.
- Model-based Attacks: Backdoor attacks inject poisoned samples that can steal sensitive data or manipulate outputs when specific keywords trigger the hidden behavior.Poisoning can occur during pre-training or fine-tuning, creating privacy leakage after deployment.
- Model-based Attacks: Model inversion reconstructs private training samples by analyzing model outputs, parameters, and gradients.Attackers use reverse engineering to recover sensitive information from private datasets.
- Model-based Attacks: Model stealing duplicates models fine-tuned on sensitive datasets by querying their responses and extracting internal information.The attacker can obtain confidential model details without direct access to the underlying dataset.
- Data-based Attacks: PCAT extracts client privacy data in Split Learning using only approximately 0.1%-5% of the private training set.The attack is described as transparent to the client and difficult to detect.
- Data-based Attacks: Special Characters Attack uses character patterns to trigger memorized raw training data from language models.The reported mechanism links uninterrupted meaningless responses with the output of memorized data.
- User-based Attacks: Membership and attribute inference attacks infer training-data membership or sensitive personal attributes from model outputs and API responses.Reported inferred attributes include personal identification details, medical records, and geographic location; MIA performance on LLMs remains unstable and largely unexplored.
Privacy Mitigation in LLMs
The survey organizes LLM privacy defenses around leakage and attacks, covering data preparation, inference protection, federated learning, differential privacy, backdoor removal, and cryptographic methods. It also highlights trade-offs including residual leakage, performance degradation, and limited scalability.
- Defense taxonomy: Defense strategies are classified into protections against privacy leakage and protections against privacy attacks.This classification organizes the mechanisms reviewed in the survey.
- Privacy leakage defenses: Data cleaning reduces privacy risk by removing or anonymizing personally identifiable information before model use.The approach includes anonymization, data minimization, and related security practices.
- Inference protection: Inference defenses include CONFAIDE, IOI, and InferDPt for assessing privacy reasoning, protecting decisions, and preserving document privacy during black-box generation.InferDPt also addresses information bias from differential privacy while enhancing remote and local model capabilities.
- Federated learning: Federated learning enables collaborative training without directly sharing distributed data, but privacy leakage and stage-specific design remain concerns.Reported leakage includes class representative reconstruction, motivating precise optimization across federated-learning stages.
- Differential privacy: Differential privacy adds random noise during processing and training to make personal details harder to infer from model training data.The survey notes that larger and more complex models may experience significant utility effects from this protection.
- Backdoor removal: Fine-tuning, FT-SAM, pruning, and fine-tuning combinations can remove or weaken backdoors, while super fine-tuning further reduces member-leakage risk.FT-SAM promotes learning of backdoor neurons, and evaluations report that pruning and fine-tuning can weaken or eliminate backdoors.
- Cryptographic protection: Cryptographic defenses provide provable security guarantees but can reduce performance, while secure multi-party computation may substantially slow Transformer inference.Obfuscation-based alternatives generally provide weaker and less predictable protection than encryption-based methods.
Practical Challenges and Future Directions
The survey identifies unresolved privacy challenges in LLM compression, risk evaluation, secure knowledge transfer, and governance, and outlines technical and interdisciplinary directions to address them.
- Privacy-Preserving Model Compression: Compression improves deployment efficiency but can expose sensitive information embedded in model weights or activations.Pruning, quantization, and knowledge distillation reduce storage and latency requirements, while distillation may transfer sensitive information through teacher outputs.
- Privacy-Preserving Model Compression: Federated learning could support collaborative, privacy-preserving compression without centralizing sensitive data.The approach aggregates model updates across clients rather than directly exposing underlying data.
- Privacy-Preserving Model Compression: Privacy-aware pruning could reduce leakage by selecting parameters or neurons according to their contribution to privacy risk.The proposed pruning strategy incorporates privacy concerns into model-parameter selection.
- Privacy Risk Evaluation: Comprehensive risk-evaluation frameworks are needed to assess leakage across training, fine-tuning, and inference, including membership and attribute inference attacks.The survey also calls for model-auditing tools within these frameworks.
- Secure Knowledge Sharing Across LLMs: Secure knowledge transfer across organizations requires methods that protect proprietary data while addressing leakage, model inversion, and unauthorized access risks.SMPC keeps combined data confidential during computation, while ZKPs verify knowledge or computation correctness without revealing the knowledge itself.
- Interdisciplinary Approaches to Privacy Governance: Privacy governance requires collaboration among AI researchers, legal experts, and policymakers, including compliance with data-erasure requirements.Benchmarks can give researchers and regulators tools to measure privacy and data-security protections in deployment.
Conclusion
The survey synthesizes LLM privacy risks, attacks, defenses, and implementation challenges. It proposes scalable, transparent, and efficient privacy solutions to guide privacy-conscious AI development.
- Conclusion: The survey reviews how model inversion, training data extraction, and membership inference can expose sensitive information from LLMs.It focuses on privacy leakage and attacks as well as the mechanisms through which models inadvertently reveal data.
- Conclusion: It categorizes privacy-preservation techniques including inference detection, federated learning, and confidential computing, evaluating their strengths and limitations.The review connects available defenses to the privacy risks they are intended to mitigate.
- Conclusion: The survey identifies practical implementation challenges and emphasizes scalable, transparent, and efficient directions for future privacy protection.Its synthesis is intended to clarify the LLM privacy landscape and guide privacy-conscious AI systems.