Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks

Erfan Shayegani; Md Abdullah Al Mamun; Yu Fu; Pedram Zaree; Yue Dong; Nael Abu-Ghazaleh

Source-linked AI summary

Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks

Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, Nael Abu-Ghazaleh

arXiv:2310.10844v1Oct 16, 2023cs.CLcs.CRcs.LG

TL;DR

Rapidly advancing and increasingly integrated LLMs require scrutiny of their security properties, especially because safety-aligned models remain susceptible to adversarial attacks. This survey synthesizes the field across learning structures, attack classes, vulnerability sources, and defenses, concluding that threats span textual, multimodal, and complex-system settings.

Problem
LLMs’ scale, generative behavior, context handling, multimodality, and integration into complex ecosystems create a need to understand their security properties and adversarial vulnerabilities.
Method
The paper systematically reviews and organizes adversarial-attack research across textual, multimodal, federated, multi-agent, and other complex LLM structures, including vulnerabilities and defenses.
Results
The survey finds that adversarial vulnerabilities arise across unimodal, multimodal, integrated, and multi-agent LLM systems, with reviewed works addressing both underlying causes and proposed defenses.
Takeaways & Limitations
Adversarial attacks must be considered across future LLM designs and the complex systems into which LLMs are integrated.
Takeaways & Limitations
Defense accuracy tends to decrease for larger adversarial sequences because longer inputs require checking more subsequences and increase mistaken harmful classifications.

Abstract

from arXiv · show

Large Language Models (LLMs) are swiftly advancing in architecture and capability, and as they integrate more deeply into complex systems, the urgency to scrutinize their security properties grows. This paper surveys research in the emerging interdisciplinary field of adversarial attacks on LLMs, a subfield of trustworthy ML, combining the perspectives of Natural Language Processing and Security. Prior work has shown that even safety-aligned LLMs (via instruction tuning and reinforcement learning through human feedback) can be susceptible to adversarial attacks, which exploit weaknesses and mislead AI systems, as evidenced by the prevalence of `jailbreak' attacks on models like ChatGPT and Bard. In this survey, we first provide an overview of large language models, describe their safety alignment, and categorize existing research based on various learning structures: textual-only attacks, multi-modal attacks, and additional attack methods specifically targeting complex systems, such as federated learning or multi-agent systems. We also offer comprehensive remarks on works that focus on the fundamental sources of vulnerabilities and potential defenses. To make this field more accessible to newcomers, we present a systematic review of existing works, a structured typology of adversarial attack concepts, and additional resources, including slides for presentations on related topics at the 62nd Annual Meeting of the Association for Computational Linguistics (ACL'24).

1 Introduction

The survey examines adversarial threats to rapidly evolving LLMs and the increasingly complex systems that integrate them. It organizes attacks by learning structure, attacker access, injection source, attack type, and end-to-end goal.

Threats and motivation: Adversarial attacks manipulate inputs to drive machine-learning systems toward reliably erroneous outputs, either targeting a specific result or merely inducing an error.Attacks may use small perturbations and differ by attacker assumptions.
Threats and motivation: LLM attacks are challenging because these models are extremely large, generative, context-aware, often multimodal, and integrated into complex ecosystems.Examples include interacting LLM agents and autonomous systems grounded on LLMs.
Scope and organization: The survey reviews attacks across text-only, multimodal, augmented, federated, multi-agent, and other complex LLM structures.Its scope includes manually generated attacks and more advanced adversarial algorithms.
Scope and organization: A particular threat model combines attacker access, injection source, attack type, and attack goals.Injection sources may include prompts, documents, or websites; goals include degrading outputs, increasing hallucinations, or bypassing alignment.
Relation to prior surveys: The survey emphasizes emerging LLMs and recent advances, contrasting its adversarial-attack focus with data-centric and earlier NLP surveys.It covers closed-source systems such as Bard and ChatGPT alongside open-source models such as Vicuna and Llama 2.

2 Background

The survey provides interdisciplinary background for readers from NLP and security communities. It covers both language-model fundamentals and adversarial attacks from a security perspective.

The background covers large language models from machine-learning and deep-learning perspectives.
It also introduces adversarial attacks from the security perspective.
The survey is designed to make interdisciplinary material accessible to researchers across NLP and security.

2.1 Language Models

This section introduces language-model structures and paradigms while identifying components that attackers could exploit. It points readers to detailed reviews of unimodal, multimodal, federated, and multi-agent models.

The survey discusses language-model structures and paradigms in parallel with its broader organization around learning structures.
It explores language-model components that could be exploited by attackers.
The cited background spans unimodal language models, multimodal models, federated large language models, and multi-agent language systems.

2.1 Language Models

The section situates language models within natural language processing and distinguishes LLMs from earlier pre-trained language models by scale, training approach, and emergent abilities. It also describes autoregressive and non-autoregressive modeling approaches.

Natural language processing includes language understanding and language generation as two central tasks.
LLMs are huge models pretrained on large amounts of data, whereas PLMs are earlier smaller pretrained models commonly fine-tuned for downstream tasks.
Emergent abilities distinguish LLMs from PLMs by enabling complex tasks absent from training data in few-shot or zero-shot settings.The passage gives in-context learning and chain-of-thought as examples associated with LLMs.
Language models assign probabilities to possible generated-text sequences through autoregressive or non-autoregressive modeling approaches.Autoregressive models typically use next-word prediction and focus on natural language generation.

2.1 Language Models

Language-model development combines data, training objectives, and capability-elicitation techniques. Pre-training commonly uses next-token prediction or denoising objectives, while instructions and in-context strategies help harness model capabilities.

Training Data: Pre-training data quantity, quality, and richness shape the outcomes of LLM training.Training aims to extract knowledge efficiently through objective functions and training strategies.
Training Strategy: Language-model pre-training commonly uses next-word prediction or denoising autoencoding objectives.Mixture-of-Denoisers is described as an advanced alternative using special tokens to select different denoising configurations.
Ability Eliciting: Ability elicitation uses task instructions or in-context learning strategies to harness LLM capabilities.The passage notes that these techniques work especially well with LLMs compared with smaller models.
Ability Eliciting: Instruction-formatted training instances contain an instruction, optional input, corresponding output, and sometimes demonstrations.Such datasets can come from annotated natural-language sources or be generated by prominent LLMs.

2.2 Security of ML Models

LLM training objectives differ from the goals of safe, useful user interaction, creating alignment challenges. Developers address these challenges through instruction tuning and reinforcement learning from human feedback.

Safety Aligned Language Models: LLMs are trained to minimize contextual word-prediction errors, whereas users want them to follow instructions usefully and safely.The mismatch contributes to instruction-following difficulties and the perpetuation of biases, toxicity, and profanity from internet data.
Safety Aligned Language Models: Instruction tuning and reinforcement learning via human feedback are used to align LLMs with desired principles.Instruction tuning fine-tunes models on instruction-based tasks, while RLHF trains reward models from human preferences.
Safety Aligned Language Models: Developers treat making LLMs both helpful and harmless as a central alignment objective.This objective is presented as a response to the gap between foundational training and user interaction goals.

2.2 Security of ML Models

Adversarial attacks are studied as a way to understand and improve model security and robustness. They manipulate inputs to induce erroneous outputs, often under constraints designed to preserve imperceptibility.

Adversarial Attacks: Adversarial attacks deliberately fool machine-learning models with carefully crafted inputs that produce unexpected outputs.The survey presents this threat as a foundation for discussing attacks against LLMs.
Adversarial Attacks: Adversarial perturbations are commonly crafted along the loss gradient to maximize their impact on network loss.Typical generation backpropagates loss to the input layer and modifies inputs within a limited noise budget.
Why study adversarial attacks?: Researchers study adversarial attacks to evaluate security and robustness and to improve models.Adversarial robustness also helps characterize worst-case behavior, while adversarial training exposes networks to adversarial examples as a defense.
Why study adversarial attacks?: Adversarial evaluation includes hostile scenarios such as evading content filters, malware detectors, or self-driving-car systems.These examples motivate examining resilience under actual or worst-case adversarial conditions.

2.2 Security of ML Models

Adversarial attacks vary by goal, model access, perturbation granularity, and generation method. In NLP, attacks modify discrete text, while gradient-based procedures iteratively construct constrained adversarial examples.

Adversarial Attack Types: Attacks may be targeted toward an attacker-chosen output or untargeted toward any erroneous prediction.Targeted attacks use the loss gradient toward the chosen target, whereas untargeted attacks follow the overall loss gradient.
Perturbation Generation: FGSM and PGD are popular methods for generating adversarial samples in deep neural networks.FGSM computes the input-loss gradient and applies a small proportional step; PGD repeatedly updates and projects perturbations.
NLP Adversarial Attacks: NLP attacks alter discrete text at the word, character, or sentence level rather than directly transferring vision perturbations.Common alterations include insertion, deletion, flipping, swapping, rearrangement, and meaning-preserving paraphrase.
Threat Models: Black-box attacks operate without internal model details, whereas white-box attacks use model parameters to backpropagate loss.Attack categories also include multi-level, character-level, word-level, and sentence-level designs.
Attack Objectives: The attack objective can combine original and adversarial losses while balancing attack effectiveness against perturbation size.λ controls the trade-off between the original loss and adversarial loss, and common adversarial losses include cross-entropy or divergence measures.
Adversarial Sample Generation: A basic untargeted attack initializes xadv from clean input, applies normalized gradient perturbations, clips values, and repeats until prediction changes.The pseudocode uses model parameters θ, clean input x, true label y, loss J, and perturbation magnitude ϵ.

3 Unimodal Attacks

This section organizes adversarial attacks on aligned unimodal LLMs into jailbreak and prompt injection attacks, comparing their assumptions, approaches, scope, and insights.

The survey reviews two prevalent attack types against aligned unimodal LLMs: jailbreak attacks and prompt injection attacks.

3.1 Jailbreak Attacks

Jailbreak research shows that aligned LLMs can leak sensitive information and produce prohibited content through manually crafted, automated, or capability-targeting attacks. The survey connects these vulnerabilities to competing objectives, mismatched generalization, and defenses that may not cover the full malicious space.

3.1.1 Initial Ad hoc Jailbreak Attempts: Even aligned LLMs remain susceptible to attacks that extract memorized sensitive and Personally Identifiable Information.Larger models’ increased memorization capacity makes privacy attacks more important to study.
3.1.1 Initial Ad hoc Jailbreak Attempts: Multi-step jailbreaking and context contamination exploit language-modeling objectives to induce ChatGPT to reveal private information.The attack constructs an adversarial context that favors accepting the malicious prompt over alignment constraints.
3.1.2 Analyzing In-The-Wild (Ad-hoc) Jailbreak Prompts and Attack Success Rates: Ad-hoc jailbreak prompts evolved toward shorter and more toxic prompts, suggesting increasingly stealthy attack strategies.
3.1.2 Analyzing In-The-Wild (Ad-hoc) Jailbreak Prompts and Attack Success Rates: 70.8%, 68.9%, 65.5%, 89.0%, and 64.8% attack success rates were reported for ChatGPT, GPT-4, ChatGLM, Dolly, and Vicuna, respectively, against in-the-wild jailbreak prompts.Dolly also reached 85.7% ASR on prohibited questions without jailbreak prompts.
3.1.3 Exploring Model Size, Safety Training, and Capabilities: Safety-training failures arise from competing objectives and mismatched generalization between complex model capabilities and simpler safety objectives.Larger models can expose attack surfaces absent in smaller models, including malicious Base64 prompts that GPT-4 can decode and accept.
3.1.3 Exploring Model Size, Safety Training, and Capabilities: Combining the two failure modes produced a 100% attack success rate against GPT-3.5-Turbo, GPT-4, and Claude v1.3.Claude’s resistance to role-play jailbreaks came with rejection of harmless role-play prompts, limiting legitimate uses.
3.1.3 Exploring Model Size, Safety Training, and Capabilities: The survey advances safety-capability parity: safety mechanisms should match the sophistication of the underlying model and cover more malicious space.
3.1.4 Automating Jailbreak Prompt Generation and Analyzing Defenses in LLM Chatbots: Automated jailbreak generation uses rephrasing, learned prompt patterns, and Reward Ranked Fine Tuning to produce additional attacks.

3.2 Prompt Injection

Prompt injection attacks manipulate LLMs into treating attacker-controlled data as instructions, enabling goal hijacking, prompt leaking, and indirect manipulation. Automated attacks can overcome application-level defenses and compromise real-world LLM-integrated applications.

3.2.1 Prompt Injection Definition, Instruction Following, Model Capabilities, and Data Safety: Prompt injection manipulates inputs so LLMs mistake attacker-controlled data for instructions and generate deceptive outputs that hijack the intended task.The underlying issue is the unclear boundary between instructions and data in LLM inputs.
3.2.1 Prompt Injection Definition, Instruction Following, Model Capabilities, and Data Safety: Larger models’ stronger instruction-following capabilities can increase susceptibility to prompt injection and jailbreak-style manipulations.The survey links this pattern to observations involving GPT-4 and Vicuna.
3.2.2 Exploring Prompt Injection Attack Variants: Goal hijacking redirects the model toward an attacker-selected objective, whereas prompt leaking seeks disclosure of the system prompt.Both are presented as principal objectives of prompt injection attacks.
3.2.2 Exploring Prompt Injection Attack Variants: LLMs may interpret injected text as instructions rather than text to process, causing them to prioritize instruction following and output protected system prompts.This behavior is illustrated through prompt-leaking and related instruction-prioritization attacks.
3.2.2 Exploring Prompt Injection Attack Variants: 28.6% prompt leaking success was lower than 58.6% goal hijacking success in Perez and Ribeiro’s study of ChatGPT attacks.Less powerful Text-Davinci-001 and Text-Curie-001 models were more resilient, likely because of weaker instruction-following capabilities.
3.2.3 System Prompt As Intellectual Property: Prompt leaking vulnerability rates were 73.1% for Vicuna-13B, 89% for GPT-3.5, and 81.9% for GPT-4.The survey reports that system prompts can be readily exposed and should not be treated as secure intellectual property.
3.2.4 Exploring Indirect and Virtual (Training Time) Prompt Injection Attacks: Virtual prompt injection can influence inference behavior after contaminating only a small fraction of instruction-tuning data, while injected prompts can also support social engineering.Examples include shifting responses about Joe Biden from 0% to 40% negative and inducing malicious or biased behavior through covert instructions.
3.2.5 Enhancing Prompt Injection Attacks: Automation and Countermeasures: An automated framework extracts application semantics, constructs framework, separator, and disruptor components, and refines the latter components using model-generated feedback.The framework masks the malicious objective while the separator breaks the prior semantic link and the disruptor performs goal hijacking.

4 Multi-Modal Attacks

Multi-modal attacks exploit text, image, and audio pathways in LLM-integrated models, including weaknesses in vision encoders and cross-modal alignment. Researchers demonstrate white-box, encoder-based, and universal attacks that can induce harmful outputs, contaminate context, or escape alignment.

4 Multi-Modal Attacks: Multi-modal LLMs accept text alongside modalities such as images, video, audio, depth, and thermal inputs.The survey examines adversarial attacks against systems including PandaGPT, LLaVA, MiniGPT-4, and related models.
4 Multi-Modal Attacks: Naive multi-modal attacks add instructions or incorrect descriptions to images to alter model behavior in classification or generation tasks.These attacks extend earlier text-based attacks that contradicted image content.
4 Multi-Modal Attacks: Vision encoders may prefer readable text over visual signals, allowing embedded text to override what the model visually observes.The survey connects this vulnerability to the optical-character-recognition abilities learned by such systems.
4 Multi-Modal Attacks: Optimized images and audio recordings provide stealthier attack inputs than directly adding text and can induce toxic content, context contamination, or alignment evasion.Several studies optimize inputs toward fixed textual targets or arbitrary behaviors.
4 Multi-Modal Attacks: White-box attacks exploit continuous image space because existing discrete-text optimization attacks may not uncover vulnerable regions in aligned models’ embedding spaces.Carlini et al. report adversarial images suggesting that such vulnerable regions may exist.
4 Multi-Modal Attacks: Multi-modal indirect prompt injections can force specified strings, including malicious URLs or instructions that condition subsequent chatbot responses.Teacher-forcing optimization is used to create outputs that function as injected instructions in later conversational context.
4 Multi-Modal Attacks: Universal adversarial inputs optimized against 66 toxic and harmful sentences generalized to broader harmful categories beyond the target corpus.Reported categories include identity attacks, disinformation, violence, and existential risks.
4 Multi-Modal Attacks: Attacks requiring knowledge only of a vision encoder can break alignment across multiple modalities, achieving alignment escape, context contamination, and hidden prompt injection.The results highlight the limitation of alignment strategies applied only to textual inputs.

5 Additional Attacks

This section surveys adversarial attacks against LLMs embedded in retrieval, database, and other integrated systems. These attacks exploit external components and can manipulate retrieved information or database operations.

Attacks on complex LLM systems target integrated retrieval, database, and other components rather than only the language model.The survey groups this emerging literature into attacks on integrated systems, multi-agent systems, and structured data.
Attacks on LLM Integrated Systems: The survey presents complex-system attacks as a relatively less-developed research direction compared with unimodal and multimodal attacks.It organizes existing work into integrated systems, multi-agent systems, and structured-data attacks.
Attacks on LLM Integrated Systems: Retrieval augmentation improves access to external information but introduces vulnerabilities when attackers manipulate retrieved content or block sources.Examples include “Arbitrarily-Wrong Summaries” and “Source Blocking,” especially consequential in medical, financial, and legal applications.
Attacks on LLM Integrated Systems: LangChain-style systems can transform user questions into SQL queries, enabling direct prompt attacks and indirect database poisoning.Attackers may read, insert, modify, or delete database data; the illustrated restricted-prompting attack deletes a table.

5.1 Adversarial Attacks In Complex Systems

Complex-system attacks extend beyond standalone LLMs to proxies, agents, federated learning, and structured-data models. The surveyed examples show that connected components create additional attack surfaces, including model manipulation and data tampering.

Attacks on LLM Integrated Systems: An LLM acting as a proxy between a victim and an attacker-controlled web service can relay harmful executable content with limited traceability.A browsing plugin can give the LLM web access; the attack dynamically generates an IP address and delivers a harmful file.
Attacks On Federated-Learning LLMs: Federated learning protects client data by aggregating locally trained models, but adversaries can alter local models or training data.The survey distinguishes adversarial and Byzantine attacks in the federated-learning setting.
Attacks On Federated-Learning LLMs: The random-mode Byzantine attack in FedMLSecurity significantly increased test loss to values ranging from 8 to 14 during training.
Attacks on Structured Data: Structured-data attacks can manipulate tables, including by replacing selected entities with adversarial entities.Tabular language models use natural-language descriptions and examples for table interpretation, creating a target for entity-swap attacks.

5.2 Earlier Adversarial Attacks In NLP

Earlier NLP adversarial attacks are organized by the granularity of text manipulation: character, word, sentence, and multi-level methods. These approaches range from simple noise and substitutions to gradient-guided and syntactically controlled generation.

Character-Level: Character-level attacks insert, delete, swap, or randomize characters, using natural spelling mistakes or synthetic noise.DeepWordBug and TextBugger modify important words in black-box or white-box settings, while punctuation and spacing changes can affect toxicity scores.
Word-Level: Word-level attacks use gradients, attention or importance scores, and synonym replacement to perturb influential words.TextFooler is an importance-based example that greedily replaces important words with synonyms.
Sentence-Level: Sentence-level attacks modify grammatically correct groups of words for tasks including inference, question answering, translation, and reading comprehension.AdvGen uses greedy loss-guided search, while SCPNs generate examples with specified syntactic structures.
Multi-Level: Multi-level attacks combine modification strategies to make adversarial changes less noticeable while increasing attack success.These schemes can use computationally intensive techniques such as FGSM and hot phrase construction.

6 Causes and Defense

The survey organizes research on vulnerability causes and defenses across LLM attacks, including textual, multimodal, and federated settings. It highlights imperfect alignment, limited adversarial exposure, censorship limits, and evolving defenses as central challenges.

Possible Causes: Insufficient exposure to diverse adversarial examples can leave models vulnerable to known and novel attacks, including prompts containing unusual or outlier words.The survey identifies broader adversarial training as a proposed response.
Possible Causes: Perfect alignment is difficult: the cited Behavior Expectation Bounds framework proves that some prompt can elicit undesirable content with probability 1.
Possible Causes: Semantic censorship can be bypassed by assembling harmful outputs from benign requests or encoding content into reversible transformations.More restrictive syntactic censorship can reduce unexpected outputs but also restrict model capacity.
Defenses: Erase-and-check defenses remove tokens and evaluate subsequences, but accuracy decreases for longer adversarial sequences because more subsequences must be checked.
Defenses: Perplexity filtering classifies prompts above a threshold as potentially harmful and can combine paraphrasing or retokenization with analysis of white-box and gray-box settings.
Defenses: Red teaming uses simulated adversarial behavior to identify harmful outputs before deployment, while language-model-based red teaming can reduce the cost and limits of manual testing.
Defenses: Adversarial training introduces perturbed examples through synonym substitutions, character manipulations, or multiple attack strategies, but universal robustness remains unresolved.
Federated Learning Settings: Federated learning systems inherit LLM vulnerabilities, while m-Krum excludes selected high-score client models before aggregation to limit malicious contributions.In an experiment with n = 7 models, m = 2 lowest-score models were aggregated, and test loss decreased across communication rounds.

7 Conclusion

The survey reviews adversarial vulnerabilities across unimodal, multimodal, and complex LLM-integrated systems, alongside their causes and defenses. It presents this systematization as a resource for understanding threats, informing alignment work, and developing protection models.

The survey covers jailbreak and prompt injection attacks against text-only LLMs, along with vulnerabilities arising in multimodal models and systems with multiple agents.
It also reviews attacks on LLM-integrated systems, including federated learning and multi-agent settings, and examines underlying vulnerability causes and proposed defenses.
Systematizing these vulnerabilities can inform alignment work and support development of protection models, while threat models provide benchmarks for evaluating security.