PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly

Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, Yue Zhang

arXiv:2312.02003v3cs.CRcs.AI

TL;DR

This survey examines how LLMs affect security and privacy through beneficial applications, offensive uses, vulnerabilities, and defenses. It finds that LLMs contribute substantially to code and data security while enabling attacks, especially user-level attacks, and identifies limited extraction-attack research and underexplored safe instruction tuning.

  • Problem

    The survey addresses the need to understand LLMs’ beneficial security and privacy applications, offensive uses, inherent vulnerabilities, and defenses.

  • Method

    The paper systematically categorizes security and privacy research on LLMs into The Good, The Bad, and The Ugly.

  • Results

    LLMs enhance code and data security, but also support offensive applications and contain vulnerabilities requiring defenses.

  • Takeaways & Limitations

    Security research should harness LLMs’ positive contributions while mitigating malicious applications and model vulnerabilities.

  • Takeaways & Limitations

    Research on model and parameter extraction attacks remains limited and primarily theoretical, partly because LLM scale and confidential model details hinder conventional attacks.

Abstract

from arXiv · show

Large Language Models (LLMs), such as ChatGPT and Bard, have revolutionized natural language understanding and generation. They possess deep language comprehension, human-like text generation capabilities, contextual awareness, and robust problem-solving skills, making them invaluable in various domains (e.g., search engines, customer support, translation). In the meantime, LLMs have also gained traction in the security community, revealing security vulnerabilities and showcasing their potential in security-related tasks. This paper explores the intersection of LLMs with security and privacy. Specifically, we investigate how LLMs positively impact security and privacy, potential risks and threats associated with their use, and inherent vulnerabilities within LLMs. Through a comprehensive literature review, the paper categorizes the papers into "The Good" (beneficial LLM applications), "The Bad" (offensive applications), and "The Ugly" (vulnerabilities of LLMs and their defenses). We have some interesting findings. For example, LLMs have proven to enhance code security (code vulnerability detection) and data privacy (data confidentiality protection), outperforming traditional methods. However, they can also be harnessed for various attacks (particularly user-level attacks) due to their human-like reasoning abilities. We have identified areas that require further research efforts. For example, Research on model and parameter extraction attacks is limited and often theoretical, hindered by LLM parameter scale and confidentiality. Safe instruction tuning, a recent development, requires more exploration. We hope that our work can shed light on the LLMs' potential to both bolster and jeopardize cybersecurity.

1. Introduction

This survey examines how LLMs affect security and privacy through beneficial applications, offensive uses, and model vulnerabilities and defenses. Its review finds predominantly positive security contributions, alongside prevalent user-level attacks and underexplored extraction and safety issues.

  • Scope and research questions: The survey investigates LLMs’ positive security and privacy impacts, offensive applications, and inherent vulnerabilities and defenses.It frames these topics as three research questions concerning benefits, threats, and weaknesses.
  • The Good: LLMs support code security across the development lifecycle and data security and privacy, with most studies reporting better performance than traditional approaches.Applications include secure coding, test generation, vulnerability detection, malicious-code detection, code fixing, and data integrity or confidentiality protection.
  • The Bad: LLMs enable hardware-, OS-, software-, network-, and user-level attacks, with user-level attacks most prevalent because of their human-like reasoning abilities.User-level examples include misinformation, social engineering, and scientific misconduct; current lack of direct OS and hardware access limits some threats.
  • The Ugly: LLM vulnerabilities include AI-model-inherent and non-AI-model-inherent weaknesses, while defenses operate through architecture, training, and inference mechanisms.Examples include data poisoning, backdoors, training-data extraction, remote code execution, prompt injection, corpora cleaning, optimization, and instruction tuning.
  • Research gaps: Model and parameter extraction research remains limited and mainly theoretical because LLM scale, private model details, and output censorship hinder conventional attacks.The survey also identifies scarce research on architecture’s impact on safety and calls for further investigation of safe instruction tuning.
  • Contributions: The survey claims a first comprehensive treatment spanning beneficial applications, offensive applications, vulnerabilities, and defenses.It reports that LLMs contribute more positively than negatively to security and privacy, while user-level attacks are most prevalent.

2. Background

LLMs extend earlier language models through transformer-based scaling, massive training corpora, and advanced pretraining. Their capabilities support applications ranging from NLP and programming to vulnerability detection and search optimization.

  • LLM evolution: LLMs evolved from statistical language models through transformer-based scaling, extensive training corpora, and advanced pretraining techniques.The paper associates this expansion with progress in AI for science and logical reasoning.
  • Model scale: LLMs typically contain hundreds of billions or more parameters trained on massive textual data.The paper links this scale to substantial advances in natural language processing.
  • Core capabilities: A capable LLM should understand natural-language context, generate human-like text, maintain contextual awareness, and solve problems.These features support tasks such as translation, text completion, article writing, and broader reasoning activities.
  • Model landscape: LLM providers include OpenAI, Google, Meta AI, Anthropic, and Cohere, with models released from 2018 to 2023.The paper notes that many models are closed-source, while BERT, T5, PaLM, LLaMA, and CTRL are open-source.

3. Overview

The paper conducts a literature review focused on LLM security and privacy, organizing collected studies into beneficial, offensive, and vulnerability-focused categories. The review finds a growing field in which positive security applications outnumber negative uses.

  • Scope: The review collates existing research to establish the state of the art and identify knowledge gaps in LLM security and privacy.Its scope concentrates on security and privacy, using GPT models as illustrative benchmarks where appropriate.
  • Research questions: The survey asks how LLMs benefit security and privacy, what risks they enable, and which vulnerabilities and defenses characterize these models.These questions correspond to the paper’s Good, Bad, and Ugly organization.
  • Collected literature: The collection contains 83 “good” papers, 54 “bad” papers, and 144 “ugly” papers.The categories represent positive contributions, attacker exploitation of LLMs, and vulnerabilities within LLMs, respectively.
  • Publication trends: October had the highest monthly publication count, with 38 papers accounting for 15.97% of the collection.Most papers were published in 2023, while 82 were released between 2007 and 2022.
  • Overall finding: LLMs are used more often to bolster security than to conduct attacks, including applications such as vulnerability detection and security test generation.The survey summarizes this pattern as LLMs contributing more positively than negatively to security and privacy.

4. Positive Impacts on Security and Privacy

LLMs support code security across the development lifecycle and contribute to data security and privacy, with reviews often reporting advantages over traditional approaches. Applications include secure coding, testing, vulnerability detection and repair, anomaly detection, phishing detection, and data protection.

  • Code Security: LLMs support the full code-security lifecycle, including coding, test-case generation, execution, monitoring, vulnerability detection, malicious-code detection, and repair.Their code-security applications span both generation and analysis tasks.
  • Code Security: ChatGPT-4.0 generated security tests that demonstrated 24 successful supply-chain attacks across 55 applications, outperforming existing security-test generators.The tests assessed the effects of vulnerable library dependencies on software applications.
  • Code Security: TitanFuzz achieved 30.38%/50.84% code coverage and detected 41 of 65 previously unknown bugs in popular deep-learning libraries.TitanFuzz uses LLM-generated input programs for fuzzing deep-learning libraries.
  • Code Security: LLM-based methods also support program repair: InferFix reported 65% to 75% accuracy, while ChatGPT-based approaches fixed 31 of 40 and 162 of 337 bugs.The reported repair approaches combine language models with static analysis or successful and failure tests.
  • Code Security: 17 of 25 studies concluded that LLM-based code-security methods outperform traditional approaches, although high false-negative and false-positive rates remain common.Reported advantages include higher code coverage, higher detection accuracy, and lower cost.
  • Data Security and Privacy: LLMs contribute to data protection through confidentiality, integrity, reliability, and anomaly-detection applications, with reviews reporting effectiveness and fewer manual interventions.Examples include semantic-anomaly monitoring, network intrusion detection, log analysis, industrial-anomaly detection, phishing detection, and GPT-generated ransomware-related GRC policies.

5. Negative Impacts on Security and Privacy

LLMs support attacks across hardware, OS, software, network, and user levels, but user-level attacks are most prevalent because of their human-like reasoning and content generation. These capabilities enable misinformation, social engineering, scientific misconduct, fraud, malware creation, and increasingly convincing phishing.

  • Attack Categories: LLM-enabled attacks span hardware-, OS-, software-, network-, and user-level positions within system infrastructure.The categories are organized according to each attack’s position within the system infrastructure.
  • Hardware-Level Attacks: LLMs can support hardware attacks by analyzing side-channel information, including AES and deep-learning accelerator scenarios.LLMs do not directly access physical devices; they analyze information associated with hardware instead.
  • Software- and Network-Level Attacks: LLMs can facilitate software attacks such as malware distribution and network attacks such as phishing, CAPTCHA bypass, and fingerprinting.Personalized AI-generated or manually designed phishing emails achieved higher click-through rates than generic messages, while ChatGPT supported phishing campaigns at scale.
  • User-Level Attacks: User-level misuse includes misinformation, social engineering, scientific misconduct, and fraud enabled by deceptive generation and inference from text.Reported examples include inferring personal attributes, extracting information from benign queries, generating difficult-to-detect misinformation, and producing fraud emails.
  • User-Level Attacks: 33 papers examined user-level attacks, making them the most prevalent category because LLMs generate human-like conversations and content.The paper attributes this prevalence partly to LLMs’ limited access to OS-level and hardware-level functionalities.

6. Vulnerabilities and Defenses in LLMs

The paper surveys privacy and security threats affecting LLMs, including inference, extraction, prompt, execution, side-channel, and supply-chain attacks, alongside defenses across training and inference.

  • AI Inherent Vulnerabilities and Threats: LLMs can infer personal information such as location, income, and sex from text with high accuracy.
  • AI Inherent Vulnerabilities and Threats: Training-data extraction attacks can retrieve individual examples from LLM training data, potentially including personal and private information.
  • AI Inherent Vulnerabilities and Threats: Prompt injections can bypass safeguards or trigger undesirable outputs, while fine-tuning can introduce backdoors through prompt attacks.
  • AI Inherent Vulnerabilities and Threats: Model and parameter extraction research remains limited and primarily theoretical because LLM scale, private ownership, and output censorship hinder conventional attacks.
  • Defenses in LLM Training and Inference: Defenses include robust training, instruction preprocessing, malicious-use detection, and generation post-processing, while non-AI vulnerabilities are highly case-specific.

7. Discussion

The discussion identifies educational and policy uses for LLMs and proposes adapting established machine-learning methods while addressing challenges unique to LLM-scale systems.

  • LLMs support cybersecurity education through software-security exercises and Capture-The-Flag activities.
  • LLMs can assist with drafting security policies, guidelines, and compliance documentation, while raising new legal and regulatory considerations.
  • Future Directions: The paper suggests applying LLMs to machine-learning-specific security tasks where traditional machine-learning methods already work.
  • Future Directions: LLMs may substitute for human effort in offensive and defensive tasks such as social engineering.
  • Future Directions: Traditional machine-learning attacks and defenses can be adapted to LLMs, including privacy-enhancing technologies such as differential privacy and federated learning.
  • Future Directions: LLM-specific extraction attacks require new approaches because parameter scale, private ownership, and confidentiality complicate traditional methodologies.

8. Related Work

Prior surveys examine selected aspects of ChatGPT and LLMs in cybersecurity, whereas this paper focuses systematically on their broader security and privacy implications.

  • Existing surveys cover LLM evolution, software engineering, medicine, and selected cybersecurity applications.
  • This paper distinguishes its focus by examining security and privacy aspects of LLMs.
  • The related work discusses practical uses such as code security, malware detection, and security-solution development but does not extensively cover the full security and privacy landscape.

9. Conclusion

The paper presents LLMs as simultaneously beneficial and risky for security and privacy, reviewing their applications, inherent vulnerabilities, and defenses to guide future use.

  • LLMs enhance code and data security while also enabling malicious applications.
  • The paper examines inherent LLM vulnerabilities and discusses defense mechanisms.
  • The survey aims to support harnessing LLM benefits while mitigating their security and privacy risks.
Loading 2312.02003v3…