PaperlayerView exact revision on arXiv

Source-linked AI summary

Security and Privacy Challenges of Large Language Models: A Survey

Badhan Chandra Das, M. Hadi Amini, Yanzhao Wu

arXiv:2402.00888v2cs.CLcs.AIcs.CR

TL;DR

LLMs provide broad capabilities but remain vulnerable to security and privacy attacks affecting models, training data, and users. This survey systematically reviews representative vulnerabilities, defenses, application-based risks, and research gaps, concluding that LLMs are highly vulnerable while mitigation techniques remain limited.

  • Problem

    LLM security and privacy vulnerabilities have not been explored sufficiently at large scale despite their broad capabilities and deployment risks.

  • Method

    The survey analyzes representative security and privacy attacks, defense mechanisms, application-based risks, research gaps, and future directions for LLMs.

  • Results

    The survey finds that LLMs are highly vulnerable to the discussed attacks and that only a limited number of mitigation techniques exist.

  • Takeaways & Limitations

    Future work should balance model utility with security and privacy protection while improving monitoring, detection, evaluation, and defense transparency.

  • Takeaways & Limitations

    Most existing attack methods target relatively small NLP models, and few have been evaluated on LLMs because access is costly and evaluation metrics are limited.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated extraordinary capabilities and contributed to multiple fields, such as generating and summarizing text, language translation, and question-answering. Nowadays, LLM is becoming a very popular tool in computerized language processing tasks, with the capability to analyze complicated linguistic patterns and provide relevant and appropriate responses depending on the context. While offering significant advantages, these models are also vulnerable to security and privacy attacks, such as jailbreaking attacks, data poisoning attacks, and Personally Identifiable Information (PII) leakage attacks. This survey provides a thorough review of the security and privacy challenges of LLMs for both training data and users, along with the application-based risks in various domains, such as transportation, education, and healthcare. We assess the extent of LLM vulnerabilities, investigate emerging security and privacy attacks for LLMs, and review the potential defense mechanisms. Additionally, the survey outlines existing research gaps in this domain and highlights future research directions.

1 INTRODUCTION

LLMs have become powerful, broadly useful language-processing systems, but their security and privacy vulnerabilities remain insufficiently explored at scale. This survey reviews those challenges, defenses, research gaps, and future directions.

  • LLMs achieve nearly human-like performance in some tasks and can adapt to new tasks through prompt engineering and in-context learning.
  • Their broad capabilities support applications including text generation, summarization, translation, and question-answering.
  • The paper identifies large-scale security and privacy vulnerabilities as an insufficiently explored area requiring in-depth study.
  • The survey comprehensively examines LLM security and privacy issues, defense mechanisms, research challenges, and future opportunities.
  • The paper compares existing surveys and research works while emphasizing recent developments, research gaps, and future directions.

2 LLM ARCHITECTURE COMPONENTS CONTRIBUTING TO VULNERABILITIES

LLMs process tokenized inputs through deep neural networks and attention mechanisms after pre-training and fine-tuning, creating multiple points where vulnerabilities can arise. The paper illustrates security attacks during training and privacy attacks during inference.

  • LLMs are pre-trained on large text datasets, fine-tuned for targeted tasks, and used to generate token sequences from contextual probability distributions.
  • Tokenization, deep neural networks, and attention mechanisms form the processing path from input text to generated output.
  • End-users, developers, training data, fine-tuning data, and deployed models can each contribute to LLM vulnerabilities.
  • In a backdoor scenario, poisoned training or fine-tuning data implants a trigger that can make the model generate harmful or inappropriate responses.
  • In a privacy attack scenario, a jailbreaking prompt can induce harmful content generation or extraction of PII and sensitive information during inference.

3 OVERVIEW OF LLM VULNERABILITIES, POTENTIAL MITIGATION, CHALLENGES AND FUTURE RESEARCH

The paper organizes LLM vulnerabilities through security and privacy perspectives, connects attack categories with mitigation strategies, and highlights shared goals across attacks. It also identifies limitations and future research needs.

  • The survey classifies LLM vulnerabilities using a goal-based approach centered on security and privacy.
  • Security categories include jailbreaking, prompt injection, backdoor, and data poisoning attacks.
  • Privacy categories include gradient leakage, membership inference, and Personally Identifiable Information leakage attacks.
  • The paper reviews representative attacks, limitations, mitigation strategies, drawbacks, common vulnerabilities, and future research directions.
  • Backdoor and poisoning attacks share the goal of causing system malfunction, while prompt injection and jailbreaking often seek sensitive information through deceptive prompts.

4 SECURITY ATTACKS OF LLMS

This section surveys security attacks against LLMs, including prompt manipulation, jailbreaking, and data or backdoor poisoning. It also reviews representative methods, attack limitations, and mitigation approaches.

  • Prompt-Based Attacks: Prompt hacking manipulates input prompts to influence LLM outputs and accomplish specific tasks.
  • Prompt-Based Attacks: Prompt injection bypasses safety filters by causing models to disregard previous instructions or perform an attacker’s intended actions.Reported consequences include data leakage, unauthorized access, hate speech, and fake-news generation.
  • Prompt-Based Attacks: HOUYI uses a pre-constructed prompt, context partition, and malicious payload to perform black-box prompt injection against LLM-integrated applications.The method reported unrestricted arbitrary LM usage and theft of application prompts on GPT-3.
  • Jailbreaking Attacks: Jailbreaking uses crafted prompts, including role-playing and DAN, to bypass safety alignment and elicit illegal, harmful, adult, or unlawful content.Its effectiveness depends strongly on prompt quality, and natural-language adaptability enables multiple prompts with equivalent semantics.
  • Jailbreaking Attacks: Automated jailbreaking methods such as GCG, GPTfuzz, and AutoDAN can achieve high success rates, although their computational cost is very high.One reported approach achieved an average success rate of 21.58%, compared with 7.33% for existing prompts.
  • Poisoning and Backdoor Attacks: Data poisoning manipulates training data to bias learning and produce incorrect outputs, while defenses include static analysis, fine-pruning, data curation, and STRIP.Advanced attacks may require carefully crafted triggers, and some methods rely on attacker-access assumptions.

5 PRIVACY ATTACKS OF LLMS

LLMs face privacy attacks that can expose training-data membership, reconstruct private data from gradients, or leak personally identifiable information. The survey reviews these attack categories, their mechanisms, defenses, and practical limitations.

  • Privacy risks: Privacy attacks exploit LLMs’ exposure to extensive training data, creating risks of memorization, data leakage, and confidential-information disclosure.Fine-tuning introduces additional privacy considerations because utility must be balanced against protecting user privacy.
  • Gradient Leakage Attack: Gradient leakage attacks infer or reconstruct private training data from gradients accessed during model training.TAG updates randomly initialized dummy data by minimizing gradient differences, using L1- or L2-norm loss with coefficient α.
  • Membership Inference Attack: Membership inference attacks determine whether a sample appeared in training data, sometimes using only model outputs rather than internal parameters.Shadow-model attacks train binary classifiers on labeled member and non-member outputs to infer membership.
  • Membership Inference Attack: MIA evaluations have focused mainly on text generation and classification, while loss-threshold approaches may provide limited discrimination.The survey notes that practical assumptions and task coverage constrain conclusions about MIA effectiveness against LLMs.
  • Privacy defenses: Existing MIA defenses include regularization, transfer learning, information perturbation, dropout, model stacking, differential privacy, and adversarial regularization.The survey cautions that proposed defenses are not sufficient across all MIA perspectives and are often evaluated on smaller language models.
  • PII Leakage Attack: PII leakage attacks can expose identifiers and sensitive records through memorization, membership inference, reconstruction, or system bugs.Sensitive PII includes names, phone numbers, addresses, SSNs, and financial or medical records.

6 DEFENSE MECHANISMS

The survey reviews defenses for prompt injection, jailbreaking, backdoor, data poisoning, gradient leakage, MIA, and PII leakage attacks. It emphasizes that many techniques remain insufficiently evaluated or effective for LLM-scale settings.

  • Defense Against Prompt Injection: Prompt-injection defenses preprocess or isolate prompts, redesign instructions, and use paraphrasing or re-tokenization to disrupt injected instructions.Detection can compare generated responses with the expected task output, while database permissions can constrain unauthorized SQL actions.
  • Defense Against Jailbreaking Attacks: Jailbreaking defenses include input/output blocking, semantic filtering, and perturbing multiple prompt copies before aggregating their outputs.The reviewed approach targets attack mitigation, non-conservatism, efficiency, and compatibility across LLMs.
  • Defense Against Backdoor Attack: Backdoor defenses use fine-tuning, pruning, activation inspection, knowledge distillation, outlier filtering, fine-pruning, and trigger classification.Some methods effective against earlier attacks may not protect against advanced attacks such as BadPrompt and BToP.
  • Defense Against Data Poisoning Attack: Data-poisoning defenses commonly rely on data validation, filtering, cleaning, anomaly detection, and limiting training epochs.The survey reports limited in-depth research for LLMs and minimal protection from filtering or reduced model capacity at the cost of test accuracy.
  • Defense Against Gradient Leakage Attack: Gradient-leakage defenses add random noise, differential privacy, or homomorphic encryption to protect model updates.Differential privacy is described as common and effective for DNNs, but it can sacrifice model utility and remains insufficiently studied for LLMs.
  • Defense Against Membership Inference Attack: MIA defenses include dropout, model stacking, differential privacy, perturbation, pruning, knowledge distillation, and regularization.Very few techniques have been proposed for LLMs, and many evaluations concern relatively small language models.
  • Defenses Against PII Leakage Attacks: PII-leakage mitigation includes deduplication and training-data curation to reduce memorized text and stored personal information.The survey identifies a need for empirical evaluation because most reviewed techniques target language models rather than LLMs specifically.

7 APPLICATION-BASED RISKS IN LLMS

LLMs create application-specific security, privacy, and social risks despite their broad usefulness. These risks include prompt-related misuse, hallucinated or biased content, cybercrime enablement, misinformation, and confidential-information leakage.

  • Human Interaction: Carefully crafted prompts can extract valuable knowledge, while reliance on prompt engineering may shift the entry barrier from coding and machine learning expertise.The paper also identifies ethical issues and limitations in human-machine interaction involving automated prompt generation.
  • Hallucination, Misinformation and Disinformation: LLMs can generate hallucinated, falsified, misinterpreted, or biased outputs, creating misinformation and disinformation risks.
  • Cybercrime and Social Issues: In governance settings, phishing, misinformation, biased content, hate speech, and confidential-information leakage can pose severe risks, including consequences for national security.
  • Science: Scientific applications face risks from hallucinations, biases, and false content, as illustrated by Galactica’s reported flaws and subsequent shutdown after public launch.

8 LIMITATIONS OF EXISTING WORKS AND FUTURE RESEARCH DIRECTION

The survey identifies dynamic, incompletely evaluated attack and defense landscapes for LLMs. It calls for broader vulnerability testing, stronger defenses, real-time monitoring, privacy-preserving techniques, and explainability balanced against new security risks.

  • Existing Attack Methods, their Limitations, and Future Research Direction: Prompt injection and jailbreaking use malicious prompts to bypass safety alignments and induce attacker-intended outputs.Jailbreaking strategies include character role-play and attention shifting, while prompts may be crafted manually or automatically.
  • Existing Attack Methods, their Limitations, and Future Research Direction: Jailbreak research is time-dependent and concentrated on ChatGPT, leaving vulnerabilities in models such as Vicuna, Bard, and Bing Chat unclear.The DAN attack used prompts collected over six months, while adversaries may continue refining prompts afterward.
  • Existing Attack Methods, their Limitations, and Future Research Direction: Most attack methods were designed for relatively small NLP models, and high commercial-model access costs limit LLM evaluations and in-depth vulnerability analysis.
  • Existing Attack Methods, their Limitations, and Future Research Direction: Future vulnerability research should adapt attacks from other domains, develop multi-objective attacks, create appropriate impact metrics, and use XAI to interpret vulnerabilities.
  • Existing Defense Mechanisms, Challenges, and Future Research Direction: ChatGPT protections can prevent single jailbreak prompts but remain vulnerable to multi-step jailbreaking, while self-reminder effects across all user queries remain difficult to evaluate.
  • Existing Defense Mechanisms, Challenges, and Future Research Direction: API-access models can use post-processing detectors to identify sensitive output and then refuse answers or mask sensitive information.
  • Existing Defense Mechanisms, Challenges, and Future Research Direction: Future defenses should balance utility with protection through flexible self-reminders, expert frameworks, real-time monitoring, robust detection, SMPC evaluation, and XAI.XAI may improve transparency but can also expose architecture, attention-based weight allocation, or other details that facilitate attacks.

9 CONCLUSION

The survey reviews LLM security and privacy challenges, attacks, defenses, application risks, limitations, and future directions. It concludes that LLMs are highly vulnerable and that mitigation techniques remain limited and require extensive evaluation and adaptation.

  • The survey analyzes LLM vulnerabilities from security and privacy perspectives alongside existing mitigation and defense strategies, including their strengths and limitations.
  • LLMs are highly vulnerable to the security and privacy attacks discussed in the survey.
  • Only a limited number of mitigation techniques are available, and solutions developed for relatively small language models require extensive evaluation and tailoring for LLMs.
  • The survey outlines research gaps, open problems, and future research directions focused on LLM security and privacy.
Loading 2402.00888v2…