PaperlayerView exact revision on arXiv

Source-linked AI summary

Security and Privacy Challenges of Large Language Models: A Survey

Badhan Chandra Das, M. Hadi Amini, Yanzhao Wu

arXiv:2402.00888v2cs.CLcs.AIcs.CR

TL;DR

Security and privacy vulnerabilities of LLMs have not been explored extensively at large scale. This survey reviews those vulnerabilities and defenses, finding high vulnerability to the discussed attacks and limited mitigation techniques.

  • Problem

    Security and privacy vulnerabilities of LLMs have not been explored extensively at large scale.

    Source: §1 · p. 2

  • Method

    The survey synthesizes LLM security and privacy challenges, vulnerabilities, mitigation and defense strategies, and research challenges and future opportunities.

    Source: §1 · p. 2

  • Results

    The survey finds LLMs highly vulnerable to the discussed attacks and reports that mitigation techniques are limited.

    Source: §9 · p. 37

  • Takeaways & Limitations

    The survey highlights research challenges and future research opportunities for secure and privacy-preserving LLMs.

    Source: §1 · p. 2

    Most surveyed attack methods target relatively small NLP models, with only a few evaluated on LLMs, and commercial access costs limit LLM attack evaluations.

    Source: §8 · p. 34

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated extraordinary capabilities and contributed to multiple fields, such as generating and summarizing text, language translation, and question-answering. Nowadays, LLM is becoming a very popular tool in computerized language processing tasks, with the capability to analyze complicated linguistic patterns and provide relevant and appropriate responses depending on the context. While offering significant advantages, these models are also vulnerable to security and privacy attacks, such as jailbreaking attacks, data poisoning attacks, and Personally Identifiable Information (PII) leakage attacks. This survey provides a thorough review of the security and privacy challenges of LLMs for both training data and users, along with the application-based risks in various domains, such as transportation, education, and healthcare. We assess the extent of LLM vulnerabilities, investigate emerging security and privacy attacks for LLMs, and review the potential defense mechanisms. Additionally, the survey outlines existing research gaps in this domain and highlights future research directions.

1 INTRODUCTION

LLMs support broad, context-adaptive tasks, but their security and privacy vulnerabilities remain insufficiently explored at scale. This survey reviews attacks, defenses, research gaps, and future directions.

1 INTRODUCTION

  • Unlike task-limited language models, LLMs handle broad tasks such as text generation, summarization, reasoning, and code generation, and can adapt through prompting and in-context learning [332],, [111], [113], [206], [278], [123].The paper argues that their security and privacy vulnerabilities have not been explored at large scale.Source: §1 · p. 2

1.1 Motivation

  • LLMs’ reliance on large Internet and annotated training datasets creates data-quality and privacy risks, while casual interactions can expose personally identifiable information [131], [23], [132].Such leakage may violate privacy laws, including HIPAA.Source: §1.1 · p. 2

1.2 Our Contributions

  • The survey systematically analyzes recent LLM security and privacy issues, representative attacks and defenses, their limitations, and research gaps and future directions.It aims to inform future attack-mitigation strategies and evaluation approaches.Source: §1.2 · p. 3

1.3 Organization

  • The paper proceeds from LLM architecture and vulnerability categories to security and privacy attacks, mitigation techniques, application-specific risks, research challenges, and its conclusion.The architecture is covered in Section 2; the survey concludes in Section 9.Source: §1.3 · pp. 3, 5

2 LLM ARCHITECTURE COMPONENTS CONTRIBUTING TO VULNERABILITIES

LLM vulnerabilities can arise across the model pipeline and its interacting users, developers, training data, and deployed model. The section illustrates how training-phase backdoors and inference-time jailbreaks can bypass safety alignment to produce harmful content or expose private information.

2 LLM ARCHITECTURE COMPONENTS CONTRIBUTING TO VULNERABILITIES

  • Transformer self-attention, pretraining and fine-tuning, RLHF, few-shot learning, and in-context learning are core components and capabilities of LLMs.Together, these capabilities have expanded language models’ potential across real-world applications.Source: §2 · pp. 5, 7
  • LLM vulnerabilities involve end-users, developers or practitioners, training or fine-tuning data, and deployed models, with attackers pursuing distinct goals and capabilities.Security attackers may seek to disrupt normal operation by eliciting harmful or inappropriate responses to benign queries.Source: §2 · pp. 5–6
  • A training-phase backdoor lets a malicious developer trigger harmful or inappropriate output by adding the token “CF” to a benign query.The backdoor may be implanted through poisoned training or fine-tuning data, and its output can bypass safety alignment.Source: §2 · p. 6, Figure 2 · p. 6
  • At inference, a malicious user can use a jailbreaking prompt to bypass safety alignment and elicit harmful content, PII, private training data, or sensitive information.The scenario involves black-box access to the model and may expose details such as email addresses and contact numbers.Source: Figure 3 · p. 6, §2 · p. 7

3 OVERVIEW OF LLM VULNERABILITIES, POTENTIAL MITIGATION, CHALLENGES AND FUTURE RESEARCH

The survey organizes LLM risks through a goal-based security-and-privacy lens, mapping attacks to mitigations and identifying limitations and future research needs.

Security risks

  • Security risks are organized into prompt hacking—jailbreaking and prompt injection—and adversarial attacks—backdoors and data poisoning.The survey adopts a goal-based approach to these security categories.Source: §3 · pp. 7–8

Privacy attacks

  • The three representative privacy attack categories are gradient leakage, membership inference, and PII leakage.Source: §3 · p. 8

Shared attack goals

  • Backdoor and poisoning attacks can share the goal of malfunctioning the AI system [300], [235], while prompt injection [238] and jailbreaking [276] can seek sensitive information through deceiving prompts [234].Source: §3 · p. 8

Mitigation strategies

  • The survey examines mitigations for attacks and for hallucination, misinformation [190], and trustworthiness [151], including self-reflection [89] and self-familiarity [154].Source: §3 · p. 8

Challenges and future research

  • It identifies future research needs in explainability, attack evaluation metrics, detection and recovery, and preserving model utility under countermeasures.The survey also analyzes shortcomings of existing attacks and countermeasures.Source: §3 · p. 8

4 SECURITY ATTACKS OF LLMS

This section surveys security attacks on LLMs, including prompt or behavior manipulation, data poisoning, and adversarial inputs, and discusses their characteristics, limitations, and potential defenses or mitigations.

4.1 Prompt Hacking.

  • Prompt hacking strategically manipulates inputs to steer an LLM toward desired or unintended outputs, and even a well-trained model can produce misleading or malicious results.It is framed as two goal-based strategies, with prompt injection described as one approach [s8.p1#fa4424, s8.p2#8873a3].Source: §4.1 · p. 9, §4.1.1 · p. 9

4.1.1 Prompt Injection.

  • Prompt injection bypasses safety filters to redirect a model’s goal or reproduce its original prompt; early work studied goal hijacking and prompt leaking [194].Methods include HOUYI’s integrated prompts and malicious payloads [146], while attack success can depend on prompt optimization and may fall under perplexity-based defenses [9].Source: §4.1.1 · pp. 9, 11, 12

4.1.2 Jailbreaking Attack.

  • Jailbreaking prompts bypass developer-imposed safety restrictions, using tactics such as role-play, attention shifting, or privilege escalation to elicit prohibited content [147].Reported approaches range from manually crafted prompts such as DAN [234] to automated attacks; EasyJailbreak reported an average violation rate of 60%, and PAIR often achieved a jailbreak in fewer than twenty queries [329] [26].Source: §4.1.2 · pp. 12–14

4.2 Adversarial Attack.

  • Adversarial attacks deliberately perturb inputs or context to mislead LLMs, producing incorrect or attacker-specified outputs.The described formulation adds a perturbation to a sample and can target either an incorrect label or a specified label.Source: §4.2 · p. 15

4.2.1 Backdoor Attack.

  • Backdoors make a model behave normally on benign inputs but malfunction when triggered, through input-, prompt-, instruction-, or demonstration-triggered mechanisms [300].Reported limits include concentration on classification tasks and small-scale models; performance on generation, translation, and large-scale LLMs remains unevaluated in much of this work [117].Source: §4.2.1 · pp. 15–17

4.2.2 Data Poisoning Attack.

  • Data poisoning inserts malicious or misleading training examples that bias model behavior, including Trojan triggers and poisoned code-completion data.Poisoning has also been demonstrated against RAG agents through knowledge-base samples [34] and vision-language models, including Shadowcast’s use of as few as 50 poisoned samples [296].Source: §4.2.2 · pp. 17–18
  • Basic poisoning attacks, including You Auto-complete Me [222], TROJANLM, and AutoPoison [240], can be mitigated with static analysis, fine-pruning [143], data curation [299], and STRIP [75].Source: §4.2.2 · p. 18
  • TrojanPuzzule [5] depends on carefully crafted triggers, AgentPoison [34] assumes potentially unrealistic black-box access, and NightShade [229] and AutoPoision [240] were evaluated on LMs or small-scale LLMs, leaving large-scale performance uncertain.Source: §4.2.2 · p. 18
  • Baseline filtering, reorganization, and rephrasing defenses [104] have been proposed, but handling adversarial inputs remains challenging.Source: §4.2.2 · p. 18
  • ChatGPT evaluation found LLMs vulnerable to word-level typo and sentence-level distraction adversarial inputs [265].Source: §4.2.2 · p. 18
  • Prompts can also be attacked; mitigation requires additional contextual information and algorithms, complicated by LLMs’ high prompt sensitivity [162].Query rephrasing [125], fine-pruning [143], and data curation [299] are potential data-poisoning defenses.Source: §4.2.2 · p. 18

5 PRIVACY ATTACKS OF LLMS

LLMs’ capacity to learn from and generate text can expose training data, user information, and PII. The section reviews gradient leakage, membership inference, and PII leakage attacks, while noting limits in defenses and evaluation.

5 PRIVACY ATTACKS OF LLMS

  • LLMs’ training-data memorization and generation capabilities create risks of disclosing sensitive information, including PII and confidential data.The section emphasizes balancing model utility with user privacy and notes that it surveys attack categories, characteristics, limitations, and potential mitigations.Source: §5 · pp. 18–19

5.1 Gradient Leakage Attack

  • TAG reconstructs up to 88.9% of tokens with 0.93 cosine similarity, while LAMP reconstructs 5× more bi-grams and 23% longer subsequences than previous methods.These gradient-based attacks target language models and require white-box gradient access; they may recover training data only at token level, and the effectiveness of proposed defenses in the LLM context remains uncertain.Source: §5.1 · pp. 19–20

5.2 Membership Inference Attack

  • MIA determines whether a sample was in a model’s training data, including through shadow-model attack classifiers and output-only access.Later approaches include likelihood-ratio membership assessment, SaMIA’s generated-sample n-gram overlap, and PREMIA’s attacks on preference data; SaMIA was tested only on classification and public WikiMIA data, while PREMIA was evaluated on open-source models.Source: §5.2 · pp. 20–22

5.3 PII Leakage Attack

  • PII leakage studies show attacks can recover user content from language-model training data and extract training examples; one GPT-2 attack extracted up to 10× more PII sequences than TAB.Although sentence-level differential privacy lowers leakage likelihood, around 3% of PII sequences still leaked. Methods remain limited in PII coverage, and performance on newer models such as Llama-3 and GPT-4 requires evaluation.Source: §5.3 · pp. 23–24

6 DEFENSE MECHANISMS

The survey reviews defenses for LLM security and privacy attacks, while noting that many techniques are inherited from language-model research and still need effective evaluation on LLMs. Across attack types, defenses face limits including advanced attacks, reduced utility, and incomplete protection.

6.1 Defense Against Security Attacks on LLMs

  • Prompt-injection defenses preprocess or re-tokenize prompts and detect compromise through prompt perplexity or response integrity, but prevalent prevention and detection can fail against optimized attacks such as JudgeDeceiver [236].For P2SQL injection, database permissions, query rewriting, and an auxiliary LLM guard can limit unauthorized SQL execution or halt suspicious results [193].Source: §6.1 · pp. 24–25
  • System-mode self-reminder reduced jailbreaking success from 67.21% to 19.34%, while SmoothLLM aggregates outputs from perturbed prompt copies to reduce attack success 100 times for Llama-2 and 50 times for Vicuna [334].Other defenses include safety filters, goal prioritization, and Llama Guard; basic mechanisms may fail against crafted attacks, and utility can suffer. Proposed defenses for multi-step jailbreaking and MasterKey remain unevaluated on those attacks.Source: §6.1 · pp. 25–26
  • Backdoor defenses include fine-tuning, pruning, activation inspection, clustering poisoned samples, and trigger detection, but black-box defenses remain scarce and methods lack evaluation on newer LLMs such as GPT-4 and Llama-3.MDP detects increased sensitivity of poisoned samples to random masking, but falls short on some models and tasks and may not stop advanced attacks such as BadPrompt [21] and BToP [295].Source: §6.1 · pp. 26–27
  • Data-poisoning defenses include filtering, cleaning, anomaly detection, epoch limits, perplexity or embedding-distance checks, fuzzing, and dataset curation, but current filtering or capacity reduction offers minimal protection at the cost of test accuracy [263].Most methods were designed for LMs, and advanced attacks such as NightShade require stronger defenses [18].Source: §6.1 · p. 27

6.2 Defense Against Privacy Attacks on LLMs

  • Gradient-leakage defenses add noise or differential privacy to updates, use encryption or clipping, harden API access, or optimize models robustly; differential privacy can reduce leaked sequences but may sacrifice utility.Existing language-domain work targets small NLP models, and further research is needed for LLM defenses.Source: §6.2 · p. 28
  • Membership-inference defenses include dropout, model stacking, differential privacy, perturbation, pruning, distillation, and regularization, but many methods lack LLM evaluation and differential privacy may impair utility.MemGuard perturbs predicted confidence scores [109], but its performance on language models remains unexplored; InferDPT applies differential privacy to black-box text generation [254].Source: §6.2 · pp. 28–29
  • PII-leakage defenses include de-duplication and curation, prompt-tuning, differential privacy (DP), API hardening, PII masking and scrubbing; DP can efficiently mitigate KART, though sanitization is incomplete and DP-SGD may be costly and reduce utility.Source: §6.2 · p. 29

7 APPLICATION-BASED RISKS IN LLMS

LLM security and privacy vulnerabilities create deployment risks that vary across application domains, from misinformation and cybercrime to transportation, healthcare, education, governance, and science. The survey also describes potential mitigations, while emphasizing that further work and safeguards are needed.

Complicated Human-Interaction

  • Prompt engineering may make valuable LLM knowledge more accessible, but raises ethical concerns and may shift the expertise barrier from coding and machine learning to prompt proficiency.The passage calls for study of the limitations of automated prompt generation through human-machine interaction.Source: §7 · p. 30

Hallucination, misinformation and disinformation dissemination

  • Hallucinations, bias, and misinformation can mislead users and harm legal, medical, political, financial, and other decisions, prompting calls for fact-checking and mitigation.Mitigations include data curation, knowledge enhancement, uncertainty measurement, retrieval, chain of verification, and self-reflection; learning-rate adjustment and synthetic rehearsal address catastrophic forgetting.Source: §7 · p. 30

Cybercrime and Social Issues

  • LLMs can enable phishing, malware, and hacking, while perpetuating social bias, representational harm, and toxic language that may incite hate or violence.The passage links these social risks largely to training corpora containing harmful language and disproportionate representation of social identities.Source: §7 · p. 31

Transportation

  • Transportation studies report bias in accident-report analysis, poor self-driving-task efficiency, and possible personal-data leaks; VistaGPT and domain-specific fine-tuning are proposed approaches.VistaGPT uses a dividing-and-recombining strategy for heterogeneous vehicle automation systems, while offline pre-trained LLM fine-tuning targets context-sensitive, safety-aware traffic tasks.Source: §7 · p. 31

Healthcare and Medicine

  • Healthcare deployment is presently inadvisable because misinformation, biased outputs, erroneous treatment advice, response variability, and sensitive-record privacy risks require human oversight,.The passage also emphasizes safety, transparency, explainability, equity, sustainability, and AI ethics.Source: §7 · p. 31

Education

  • In education, ChatGPT can propagate inaccurate or imprecise concepts and misconceptions, so its use warrants supervision and verification [176],,.Excessive dependence by educators and learners may compromise learning, and the passage cites scientific misconduct as an example of concern [176].Source: §7 · p. 31

Governance and Science

  • Governance risks include AI-enabled phishing and confidential-information leakage that can threaten national security [88], while science faces hallucinations, bias, opaque reliability, and impaired peer review,,, [16].The Galactica model was shut down after public launch following reported flaws; the passage calls for clear principles to guide AI use in scientific exploration,, [16].Source: §7 · pp. 31–32

8 LIMITATIONS OF EXISTING WORKS AND FUTURE RESEARCH DIRECTION

The survey reviews LLM security and privacy attacks, defenses, their limitations, and future research directions. It highlights gaps in attack and defense coverage, evaluation, and explainability.

Existing Attack Methods, their Limitations, and Future Research Direction

  • Jailbreak research is ChatGPT-centered and prompt strategies evolve, while automated attacks may underperform manual prompts and backdoor studies mostly target classification.The DAN prompt collection may not capture later optimized prompts; vulnerabilities in other LLMs and backdoors on question answering, summarization, and translation remain uncertain or underexplored.Source: §8 · pp. 32–33
  • Privacy-attack capability depends on access, but white-box assumptions can be impractical and language-model evidence is limited; heuristic benchmark data may also be biased or noisy.White-box access can enable recovery of more private training instances than black-box access, while ProPILE’s heuristic collection may introduce bias, disassociation, or noise.Source: §8 · pp. 33–34
  • Many attacks target small NLP models, and LLM vulnerability analysis lacks comprehensive metrics; future work should adapt attacks, develop novel methods, and improve evaluation.Commercial access costs and federated-learning compute demands constrain evaluation; hallucination research also needs to examine safety, including abstention from harmful content.Source: §8 · p. 34

Existing Defense Mechanisms, Challenges, and Future Research Direction

  • Existing defenses remain vulnerable to multi-step jailbreaks and evolving backdoors, while some privacy protections trade model utility for security.The survey calls for flexible self-reminding defenses, broader task and few-shot testing, dynamic defenses, real-time privacy monitoring, and evaluation of secure multi-party computation while balancing utility and protection.Source: §8 · pp. 35–36

Role of explainable AI in Enhancing Security and Privacy of LLMs

  • XAI can clarify LLM decisions and influential input features, supporting trust, error analysis, and bias detection; LIME [214] and SHAP [153] explain behavior without internal access.The survey identifies interpretability methods as important for making LLM use more reliable and trustworthy.Source: §8 · p. 36
  • XAI may expose model internals and facilitate privacy or extraction attacks, yet its security implications—including prompt injection and jailbreaking in LLMs—remain insufficiently explored.The survey notes that XAI-related vulnerabilities have mostly been studied in DNNs; LLM-specific interpretation and risk mitigation require further research.Source: §8 · pp. 36–37

9 CONCLUSION

The survey finds LLMs highly vulnerable to security and privacy attacks, while available mitigation techniques are limited. It reviews existing defenses and their limitations, identifies research gaps, and outlines future directions for secure and privacy-preserving LLMs.

9 CONCLUSION

  • The survey finds LLMs highly vulnerable to the discussed security and privacy attacks.It analyzes vulnerabilities across both security and privacy aspects.Source: §9 · p. 37
  • Mitigation techniques are limited, and defenses developed for smaller language models require extensive evaluation and tailoring for LLMs.The survey reviews existing mitigation strategies and their strengths and limitations.Source: §9 · p. 37
  • The survey identifies research gaps, open problems, and future directions focused on LLM security and privacy.Its overarching goal is to support reliable and useful LLMs through resolving vulnerabilities and advancing secure, privacy-preserving systems.Source: §9 · p. 37

Related papers

Loading 2402.00888v2…