Exploring Vulnerabilities and Protections in Large Language Models: A Survey

1 Introduction

• Safety controls coexist with risks that LLMs may teach malicious skills, while attack strategies continue to evolve in a “shield and spear” dynamic.The introduction cites risks including breaking into vehicles or synthesizing harmful substances.Source: §1 · p. 1
• Interaction-layer attacks can target open- and closed-source LLMs without access to model internals, making them practical across deployments.The survey focuses on Prompt Hacking and Adversarial Attacks; gradient-based attacks requiring architecture or gradient access are not feasible against closed-source models.Source: §1 · pp. 1–2
• These attacks can cause sensitive-data leakage, unauthorized access, misinformation, and harmful content, with impacts amplified by widespread LLM deployment.LLMs trained on massive, uncurated internet data may contain sensitive information such as medical reports or government IDs.Source: §1 · pp. 1–2
• The survey aims to clarify vulnerabilities and defenses across model types as groundwork for broader security research and more resilient AI systems.Figure 1 outlines attack categories and corresponding defense mechanisms.Source: §1 · pp. 1–2, Figure 1 · p. 2

2 Prompt Hacking

• Prompt hacking crafts and manipulates inputs to steer LLM outputs, and instruction-tuned models can be vulnerable to it.The section identifies prompt injection and jailbreaking as its two main strategies.Source: §2 · p. 2

2.1 Prompt Injection

• Prompt injection can make a model disregard initial instructions, leading to data leakage, unauthorized access, hate speech, fake news, and other breaches.Attackers bypass filters by using precisely formulated prompts.Source: §2.1 · pp. 2–3
• Compositional Instruction Attacks embed harmful instructions in benign prompts through conversational-persona or narrative-writing transformations, with high success rates across evaluated LLMs.The study reports that models struggle to identify malicious intent in multi-intent instructions and are vulnerable to repetitive attacks.Source: §2.1 · p. 3

2.1.1 Defense Against Prompt Injection

• Prompt-injection defenses include prevention through paraphrasing, re-tokenization, isolation, or redesigned instructions, and detection through responses or prompt perplexity.Perplexity detection is less effective when injected and desired tasks are of the same type.Source: §2.1.1 · p. 3

2.2 Jailbreaking Attack

• Jailbreaking bypasses developer-imposed constraints; strategies include role-play, attention shifting, and privilege escalation to elicit restricted behavior.Figure 2 depicts GPT-4o jailbreaking using a role-play strategy inspired by Learn Prompting.Source: §2.2 · p. 3, Figure 2 · p. 4
• MasterKey reports a 21.58% jailbreak success rate versus 7.33% for existing prompts; other approaches include genetic-algorithm prompts, visual attacks, and PAIR, which can jailbreak in fewer than twenty queries.The genetic-algorithm approach does not require access to model architecture or parameters.Source: §2.2 · pp. 3–4

2.2.1 Defense Against Jailbreaking Attack

• Jailbreak defenses use updated filters, preprocessing, input/output blocking, semantic filtering, self-reminders, and red-flag keyword blocking, but basic mechanisms may not suffice against privilege escalation.The section calls for more advanced defenses against sophisticated attacks.Source: §2.2.1 · pp. 4–5, §2.2 · p. 4
• SmoothLLM perturbs multiple prompt copies and aggregates their outputs, while DiffPure addresses visual jailbreak examples; defenses must balance security with model utility.SmoothLLM is described as efficient and compatible across different LLMs.Source: §2.2.1 · p. 5

3 Adversarial Attack

• Adversarial attacks manipulate inputs to deceive models and can occur during training or inference, unlike prompt hacking, which targets inference-time prompts.Source: §3 · p. 5

3.1 Backdoor Attack

• Backdoors hide triggers during training so models behave maliciously on specific inputs while otherwise functioning normally; variants use input, prompt, instruction, or demonstration triggers.Such attacks are particularly challenging to defend against when models come from untrusted sources.Source: §3.1 · pp. 5–6
• ProAttack achieved nearly 100% success across various text-classification tasks, while CBA achieved a 100% attack success rate on LLaMA-7B with minimal impact on model accuracy.ProAttack uses clean-label prompt triggers without external markers; CBA activates only when all distributed trigger keys appear simultaneously.Source: §3.1 · pp. 5–6

3.1.1 Defense against Backdoor Attack

• Backdoor defenses include fine-tuning, embedding purification, and CUBE clustering to identify poisoned samples.Fine-mixing combines backdoored and pretrained weights before refining them on a small clean dataset.Source: White-Box Defense Strategies · p. 6, §3.1.1 · p. 6

White-Box Defense Strategies

• Black-box backdoor defenses use rare-word perturbations, sentence perplexity, or random masking to detect triggers through changes in model outputs or probabilities.MDP is described as particularly effective in few-shot learning scenarios.Source: White-Box Defense Strategies · p. 6

3.2 Data Poisoning Attack

• Data poisoning alters training data, often by inserting trigger phrases that induce manipulated predictions or systemic errors.The attacks are especially effective when training data comes from external or unverified sources.Source: §3.2 · p. 6
• Even a small number of poisoned examples can produce consistently negative or flawed outputs across tasks, and larger language models are more susceptible.Data filtering and reducing model capacity offer limited protection and can reduce model performance.Source: §3.2 · p. 7

3.2.1 Defense Against Data Poisoning Attack

• Poisoning defenses include BERT-embedding anomaly detection and dataset cleaning, but existing filtering or capacity reduction provides minimal protection at a test-accuracy cost.Dataset cleaning can remove near-duplicate poisoning samples, known triggers, and payloads.Source: §3.2.1 · p. 7

• The survey covers prompt injection, jailbreaking, backdoors, and data poisoning, and concludes that evolving attacks call for continued research and robust, adaptive defenses.Source: §4 · p. 7