Source-linked AI summary
Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models
Arijit Ghosh Chowdhury, Md Mofijul Islam, Vaibhav Kumar, Faysal Hossain Shezan, Vaibhav Kumar, Vinija Jain, Aman Chadha
TL;DR
LLMs’ growing use across sectors makes their vulnerabilities a concern for system security and trustworthiness. This paper surveys attack mechanisms and defenses, organizing attacks into a taxonomy and examining reported outcomes. It concludes that LLMs face diverse threats and that mitigation approaches vary in applicability across models, contexts, and attacks.
Problem
LLMs are increasingly integrated across sectors but remain susceptible to attacks that challenge their integrity and reliability.
Source: §1 · p. 1Method
The paper surveys attack mechanisms and mitigation approaches, proposes a taxonomy, and discusses implementation details, challenges, and future directions.
Source: §1 · p. 1Results
The survey finds diverse threats to LLM security and reliability and describes mitigation approaches including filtering, guardrails, robust training, adversarial training, and safety context distillation.
Source: §6 · p. 8Takeaways & Limitations
The survey concludes that mitigation effectiveness may vary across models, contexts, and specific attacks, limiting the universal applicability of some defenses.
Source: §7 · p. 9The effectiveness of discussed mitigations varies across models, contexts, and attacks, and some defenses have limited universal applicability.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Large Language Models (LLMs) have become a cornerstone in the field of Natural Language Processing (NLP), offering transformative capabilities in understanding and generating human-like text. However, with their rising prominence, the security and vulnerability aspects of these models have garnered significant attention. This paper presents a comprehensive survey of the various forms of attacks targeting LLMs, discussing the nature and mechanisms of these attacks, their potential impacts, and current defense strategies. We delve into topics such as adversarial attacks that aim to manipulate model outputs, data poisoning that affects model training, and privacy concerns related to training data exploitation. The paper also explores the effectiveness of different attack methodologies, the resilience of LLMs against these attacks, and the implications for model integrity and user trust. By examining the latest research, we provide insights into the current landscape of LLM vulnerabilities and defense mechanisms. Our objective is to offer a nuanced understanding of LLM attacks, foster awareness within the AI community, and inspire robust solutions to mitigate these risks in future developments.
1 Introduction
LLMs have brought major advances in language tasks but remain vulnerable to attacks that threaten their integrity and reliability. The paper surveys attack mechanisms and consequences, proposes a taxonomy, and reviews mitigations and future research directions.
1 Introduction
- The survey examines attacks on LLMs, their mechanisms, consequences, and the evolving threat landscape.It also dissects adversarial techniques and discusses limitations of current defenses and potential future research avenues.Source: §1 · p. 1
- LLM vulnerabilities matter because these systems are integrated across sectors, making their security and trustworthiness consequential.The survey categorizes attacks by access to model weights and attack vectors, which present distinct challenges.Source: §1 · p. 1
OUR CONTRIBUTIONS
- The paper proposes a novel taxonomy of LLM attacks to structure the research landscape and help researchers identify areas of interest.It also details attack and mitigation approaches, including implementation details, and discusses challenges and future directions.Source: OUR CONTRIBUTIONS · p. 1
2 Exploring LLM Security: White and Black Box Attacks
The section distinguishes attacks with full access to model internals from attacks operating through limited input-output access. It describes different capabilities and risks associated with each setting.
2 Exploring LLM Security: White and Black Box Attacks
- The section frames security analysis across white- and black-box perspectives as important for understanding and protecting against complex threats.The two perspectives differ in the degree of access to model internals.Source: §2 · pp. 1–2
2.1 White Box
- White-box attacks use access to model architecture, training data, and algorithms to extract sensitive information, manipulate outputs, or insert malicious code.Examples discussed include context contamination and prompt injection aimed at changing outputs or reducing quality.Source: §2.1 · p. 2
2.2 Black Box
- Black-box attacks exploit model vulnerabilities through the input-output interface, a setting described as realistic in practical scenarios.Reported risks include sensitive-data extraction, biased outputs, and reduced trust; cited work examines jailbreaks and API-based attacks.Source: §2.2 · p. 2
3 LLM Attacks Taxonomy
The taxonomy surveys jailbreaks, prompt injections, and training-data attacks, showing how adversaries can manipulate model behavior or expose sensitive information. It also summarizes reported attack approaches and mitigation-related findings across these areas.
3.1 Jailbreaks
- Jailbreaks include iterative query refinement, trigger-word and nested-prompt techniques, multimodal or low-resource-language inputs, and automated character-sequence attacks.PAIR uses an attacker LLM to refine candidate semantic jailbreaks and can often produce one in under twenty queries; reported attacks also include translating unsafe English inputs into low-resource languages to circumvent GPT-4 safeguards.Source: §3.1 · pp. 2–3
- The taxonomy figure groups prevalent attacks under jailbreaks, prompt injection, and data poisoning, while dividing mitigation efforts into training-based and moderation-based approaches.Its branches highlight representative works for each theme.Source: Figure 1 · p. 3
3.2 Prompt Injection
- Prompt-injection research also covers goal hijacking, prompt leaking, malicious-content generation, training-data manipulation, and prompt-optimization frameworks.AutoDAN uses handcrafted jailbreak prompts to initialize hierarchical genetic search and employs sentence- and word-level crossover policies; its method achieves lower sentence perplexity.Source: §3.2 · pp. 3–4
- HOUYI achieved an 86.1% attack success rate across 36 real-world LLM-integrated services by iteratively refining injected prompts.Its context-inference, payload-generation, and feedback phases aim to make malicious payloads appear to be questions rather than data; reported consequences include unauthorized service imitation and computational-power exploitation.Source: §3.2 · p. 3
3.3 Data Poisoning
- Fine-tuning GPT-3.5 on 10 PII instances enabled disclosure of 650 of 1,000 target PIIs, versus 0 without fine-tuning; Janus increased disclosure to 699.The analysis reports stronger memorization and PII recovery for larger models and real training data, and greater effectiveness from fine-tuning than prompt engineering alone.Source: §3.3 · p. 4
- Data-poisoning studies cover safety risks from fine-tuning and instruction tuning, as well as methods for filtering unsafe examples and mitigating backdoors.Reported work includes safety tuning to balance safety and performance, ForgetFilter to improve safety without sacrificing performance, and Autopoison, which alters behavior without semantic degradation.Source: §3.3 · pp. 4–5
4 Human Interference
The section contrasts human-crafted and automated adversarial attacks, then surveys external filtering and internal training-based defenses. It highlights trade-offs in scalability, maintenance, and deployment constraints.
4.1 Human Red Teaming
- Manual adversarial prompting relies on human expertise to identify model vulnerabilities, though it is described as slow and not scalable.Curated prompts have been tested across multiple models, and crowdsourced sites collect jailbreak prompts for analysis.Source: §4 · p. 5, §4.1 · p. 5
- Human red-teaming ranges from curated harmful prompts to crowdsourced and interactive adversarial-example generation.Two prompts drawn from a collection of 6,387 malicious prompts achieved 99% attack success on GPT-3.5 and GPT-4; other approaches use interfaces to help participants craft and label adversarial examples.Source: §4.1 · p. 5
4.2 Automated Adversarial Attacks
- Automated adversarial methods scale attack generation through search, optimization, and model-assisted prompt creation; some suffixes transfer to black-box production LLMs.Examples include MASTERKEY, which fine-tunes another LLM on jailbreak prompts, and a universal method combining greedy and gradient-based search; AutoDAN uses hierarchical genetic search, while ARCA uses coordinate ascent.Source: §4.2 · pp. 5–6
4.3 Mitigation Strategies
- Mitigation strategies are organized by their deployment strategy, with guarding presented as an external alternative to model retraining.The section further distinguishes external input/output filtering from internal training and fine-tuning approaches.Source: §4.3 · p. 6, §4.3.1 · p. 6, §4.3.2 · p. 7
4.3.1 External: Input/Output filtering or Guarding
- External defenses filter adversarial inputs or anomalous outputs without retraining, using feature-based detection or classifiers to distinguish unsafe prompts.Feature-based methods can avoid extensive retraining, while classifier approaches offer a more nuanced distinction but require maintenance; filtering can alter intended outputs or flag legitimate queries, and perplexity-based methods may not work for black-box models.Source: §4.3.1 · pp. 6–7
- External safeguards for manual jailbreaks include taxonomy-based Llama Guard, programmable NeMo-Guardrails, and secondary-LLM output filtering.The described approaches face constraints including chain-of-thought prompting scalability, language compatibility, and operational costs.Source: §4.3.1 · p. 7
4.3.2 Internal: Model training/fine-tuning
- Internal safety methods include supervised safety fine-tuning, RLHF with a safety reward model, and safety context distillation.The survey reports that RLHF has been shown to make models more robust to jailbreak attempts; context distillation removes the safe persona prompt after fine-tuning to distill that context into the model.Source: §4.3.2 · p. 7
5 Challenges and Future Research
The section identifies future research needs in monitoring, multimodal defenses, attack benchmarking, and explainability. These directions address detection, safety evaluation, and interpretation challenges for LLMs.
5.1 Real-time Monitoring Systems
- Future monitoring systems should scrutinize outputs for undesirable content and remain resilient to adversarial evasion.The section notes that current evaluation mechanisms are inadequate to detect threats such as data exposure, misinformation, illegal content, and assistance with criminal activities.Source: §5.1 · p. 7
5.2 Multimodal Approach
- Multimodal defenses should improve input sanitization and validation and use custom prompts to help prevent jailbreaks.The section frames multimodal capabilities as both an opportunity and a safety challenge.Source: §5.2 · pp. 7–8
5.3 Benchmark
- A scalable, near-real-time benchmark infrastructure is needed to compare attack efficacy and better support practical deployment.The section says existing benchmarking frameworks are often insufficient for real-world scenarios.Source: §5.3 · p. 8
5.4 Explainable LLMs
- Explainable LLM research should make model decisions more interpretable while addressing accuracy, accessibility, privacy, and data-security challenges.The section highlights the difficulty of reliably attributing decisions in ways that are accurate and accessible to non-experts.Source: §5.4 · p. 8
6 Conclusion
The conclusion presents a taxonomy and overview of LLM attacks and mitigation strategies. It emphasizes that diverse threats challenge security and reliability in real-world applications.
6 Conclusion
- LLMs face diverse threats that challenge their security and reliability in real-world applications.The paper organizes attack literature into a novel taxonomy to provide structure and aid future research.Source: §6 · p. 8
- Mitigation strategies discussed include data filtering, guardrails, robust training, adversarial training, and safety context distillation.Source: §6 · p. 8
- The conclusion calls for continued work on attack detection, mitigation, and model resilience to address security risks.Source: §6 · p. 8
7 Limitations
The paper notes limitations in survey coverage, mitigation generalizability, ethical scope, the changing threat landscape, and practical defense implementation.
7 Limitations
- Rapid advances in LLMs and attack methods mean some emerging threats may not be covered.New vulnerabilities could also emerge after publication.Source: §7 · pp. 8–9
- Mitigation effectiveness may vary across models, contexts, and specific attacks, limiting universal applicability.Source: §7 · p. 9
- The survey focuses primarily on technical security, leaving broader ethical and societal implications less comprehensively explored.The paper notes that dual-use technologies require ethical consideration beyond its scope.Source: §7 · p. 9
- Attackers continually adapt strategies in response to defenses, so the paper captures only a snapshot of an ongoing threat landscape.The paper says continuous research and vigilance are needed to address this adaptive threat environment.Source: §7 · p. 9
- Practical defense implementation faces computational overhead, scalability, and ongoing-update challenges, while balancing security and usability remains underexplored.Source: §7 · p. 9
A Appendix
The appendix table summarizes attacks targeting LLMs by category, threat model, attack strategy, and evaluated models.
A Appendix
- The table organizes surveyed attacks into jailbreak, prompt injection, and data poisoning categories.Source: Table 1 · p. 13
- For each listed paper, the table outlines its threat model and attack strategy.Source: Table 1 · p. 13
- The table also lists the LLM models evaluated in each paper.Source: Table 1 · p. 13