Source-linked AI summary
Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey
Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, Yu Qiao
TL;DR
Conversational LLMs can be misused to generate harmful responses, while existing surveys lack a unifying overview of conversation-safety methods. This survey systematically reviews attacks, defenses, and evaluations, concluding that the field remains scope-limited and faces unresolved challenges including limited attack diversity, false refusal, and non-unified evaluation standards.
Problem
Existing surveys address social impacts or specific method subsets but lack an integrated overview of attacks, defenses, and evaluations for LLM conversation safety.
Method
The survey synthesizes inference-time and training-time attacks, alignment, inference guidance, filters, safety datasets, and evaluation metrics.
Results
The survey provides a structured overview of attacks, defenses, and evaluations for LLM conversation safety.
Takeaways & Limitations
The taxonomy is intended to support understanding of LLM conversation safety and future research toward socially beneficial LLMs.
Takeaways & Limitations
The survey is limited in scope because it focuses on LLM conversation safety, while the field also faces limited attack-domain diversity, false refusal, and a lack of unified evaluation standards.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are now commonplace in conversation applications. However, their risks of misuse for generating harmful responses have raised serious societal concerns and spurred recent research on LLM conversation safety. Therefore, in this survey, we provide a comprehensive overview of recent studies, covering three critical aspects of LLM conversation safety: attacks, defenses, and evaluations. Our goal is to provide a structured summary that enhances understanding of LLM conversation safety and encourages further investigation into this important subject. For easy reference, we have categorized all the studies mentioned in this survey according to our taxonomy, available at: https://github.com/niconi19/LLM-conversation-safety.
1 Introduction
LLM conversation safety addresses the risk that conversational models may facilitate harmful activities and produce toxic, discriminatory, or misleading content. This survey unifies research on attacks, defenses, and evaluations into a comprehensive overview.
- Conversational LLMs can facilitate fraud and cyberattacks while propagating toxic content, discriminatory biases, and misinformation.
- Research on conversation safety has expanded around ensuring that LLM responses are free from harmful information.
- The study focuses on autoregressive conversational LLMs, including pre-trained models such as Llama-2 and GPT-3 and fine-tuned models such as Llama-2-chat, ChatGPT, and GPT-4.
- Existing surveys often address social impacts or narrow method subsets without integrating attacks, defenses, and evaluations.
- The survey covers inference-time and training-time attacks, safety alignment, inference guidance, filtering, safety datasets, and evaluation metrics.
2 Attacks
The survey organizes attacks into inference-time prompt attacks and training-time attacks that alter model behavior or weights. It describes red teaming, jailbreak prompts, neural prompt modification, poisoned data, and backdoors as attack mechanisms.
- Inference-Time Attacks: Inference-time attacks use adversarial prompts to elicit harmful outputs without modifying model weights, whereas training-time attacks influence model weights through training data.
- Red-Team Attacks: Red-team attacks generate malicious instructions representing common user queries, either through human crowdworkers or another LLM.
- Jailbreak Attacks: Because built-in security can resist basic red-team prompts, template-based and neural prompt-to-prompt attacks enhance or iteratively modify instructions to jailbreak LLMs.
- Template-Based Attacks: Heuristic templates use explicit instruction-following or implicit domain shifts, while optimization-based methods search for token-level triggers or natural-language templates.
- Training-Time Attacks: Training-time attacks can use a small amount of poisoned data to significantly compromise safety capabilities, including in models that underwent safety alignment.
- Training-Time Attacks: Emulated disalignment shows that adversarial training can be reproduced by sampling from open-source models at inference time, making fine-tuning attacks more distributable.
- Training-Time Attacks: Backdoor attacks insert triggers into training data so models behave normally on benign inputs but unsafely when the trigger appears.
3 Defenses
The survey organizes LLM defenses into internal safety alignment, inference-time guidance, and input/output filtering. These layers include training methods, prompt or token-selection guidance, and rule-based or model-based filters.
- Defense framework: The hierarchical framework places safety alignment at the innermost layer, inference guidance in the middle, and filters at the outermost layer.The middle and outermost layers operate at inference time.
- Safety alignment: Safety alignment fine-tunes models to improve internal safety, using methods such as SFT, RLHF, DPO, and multi-objective optimization.Multi-objective methods support trade-offs among safety, helpfulness, and honesty.
- Safety alignment: Alignment data includes single-answer demonstrations for SFT and ranked multiple-answer preferences for methods such as DPO.Preference data conveys safety knowledge through the partial ordering of answers.
- Inference guidance: Inference guidance improves safety without changing model parameters through system prompts and token-selection methods such as RAIN.RAIN searches possible continuations, scores their safety, and aggregates scores during backward selection.
- Filtering: Filters detect harmful inputs or outputs using rule-based or model-based approaches, including perplexity checks, perturbations, classifiers, and LLM-based detection.NeMo Guardrails provides a customizable platform for deploying such controls.
4 Evaluations
The survey evaluates attacks and defenses through red-team datasets, optional jailbreaks, defended LLM outputs, and evaluation metrics. Datasets vary in topics, formulations, and languages, while metrics assess attack success and finer properties such as robustness and efficiency.
- Evaluation pipeline: The evaluation pipeline runs red-team datasets through optional jailbreak attacks and defended LLMs before analyzing the resulting outputs.The survey covers both evaluation datasets and evaluation metrics.
- Evaluation datasets: Safety datasets vary by size, topics such as toxicity and discrimination, formulations, and languages including English and Chinese.Forms include red-team statements, questions, Q&A pairs, preferences, and dialogue.
- Evaluation datasets: Datasets support multiple tasks through harmful statements, questions, Q&A pairs, ranked preferences, multi-turn dialogue, and jailbreak templates.These formats enable completion, response-generation, preference, and more difficult jailbreak evaluations.
- Evaluation metrics: Attack success rate measures whether an attack elicits harmful content using manual review, references, keywords, LLM judgments, or toxicity classifiers.These approaches differ in how they identify successful attacks, especially when refusals are implicit.
- Evaluation metrics: Fine-grained metrics examine attack robustness, false positives, and efficiency, but no standardized quantitative method currently measures attack efficiency.ROGUE and BLEU can compare outputs with references, while token-level optimization and LLM-based methods differ in speed.
5 Conclusion
The survey synthesizes research on LLM conversation-safety attacks, defenses, and evaluations, while identifying unresolved challenges in attack diversity, defense helpfulness, and evaluation standardization. It presents this focused overview as a contribution to developing socially beneficial LLMs.
- Conclusion: The survey covers inference-time and training-time attacks, alignment, inference guidance, filters, safety datasets, and evaluation metrics.Its scope is organized around attacks, defenses, and evaluations for LLM conversation safety.
- Challenges: Limited attack-domain diversity can make attacks vulnerable to retrospective defenses based on domain-aligned data.The survey specifically identifies fixed templates and optimization paradigms as sources of this limitation.
- Challenges: Over-alignment or inaccurate filtering can cause false refusals that reduce helpfulness when safe questions are treated as dangerous.The survey labels this problem false refusal or exaggerated safety.
- Challenges: Nonstandard evaluation criteria hinder comparisons because ASR results can vary with GPT versions and evaluation prompts.The survey calls for unified evaluation standards and metrics.