Source-linked AI summary
GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis
Yueqi Xie, Minghong Fang, Renjie Pi, Neil Gong
TL;DR
Jailbreak prompts threaten LLM safety, while existing detectors can require extensive data collection and training. GradSafe compares safety-critical gradients from prompts paired with compliance responses, and the authors report that its Llama-2 variants outperform finetuned and moderation-based detectors on the evaluated benchmarks.
Problem
Jailbreak prompts threaten LLM safety, while moderation APIs and finetuned LLM detectors have limits including general-toxicity focus, overestimation of safety risks, and resource-intensive training.
Source: §1 · pp. 1–2Method
GradSafe compares gradients from a prompt paired with a compliance response against unsafe-gradient patterns on safety-critical parameters, using zero-shot thresholding or logistic-regression adaptation.
Source: §1 · p. 2Results
On ToxicChat and XSTest, Llama-2-based GradSafe-Zero without further training surpasses Llama Guard and leading online moderation APIs, while GradSafe-Adapt improves on ToxicChat.
Source: §1 · p. 2Takeaways & Limitations
Restricting detection to safety-critical parameters reduces the storage and computation required compared with storing full LLM gradients.
Source: §4.4.1 · p. 8GradSafe assesses prompt safety comprehensively but does not provide fine-grained classification of specific classes.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Large Language Models (LLMs) face threats from jailbreak prompts. Existing methods for detecting jailbreak prompts are primarily online moderation APIs or finetuned LLMs. These strategies, however, often require extensive and resource-intensive data collection and training processes. In this study, we propose GradSafe, which effectively detects jailbreak prompts by scrutinizing the gradients of safety-critical parameters in LLMs. Our method is grounded in a pivotal observation: the gradients of an LLM's loss for jailbreak prompts paired with compliance response exhibit similar patterns on certain safety-critical parameters. In contrast, safe prompts lead to different gradient patterns. Building on this observation, GradSafe analyzes the gradients from prompts (paired with compliance responses) to accurately detect jailbreak prompts. We show that GradSafe, applied to Llama-2 without further training, outperforms Llama Guard, despite its extensive finetuning with a large dataset, in detecting jailbreak prompts. This superior performance is consistent across both zero-shot and adaptation scenarios, as evidenced by our evaluations on ToxicChat and XSTest. The source code is available at https://github.com/xyq7/GradSafe.
1 Introduction
Jailbreak prompts threaten LLM safety, while existing moderation and finetuned-detector approaches have limits. GradSafe uses safety-critical gradients to detect them without further LLM finetuning, and its zero-shot and adapted variants outperform comparison systems in the reported benchmarks.
1 Introduction
- Undetected jailbreak prompts in customization training data can let a model be finetuned to comply with them, creating a risk of unsafe behavior.Jailbreak prompts may also facilitate misuse of LLMs.Source: §1 · p. 2
- Existing moderation APIs target general toxicity, zero-shot LLM detectors can overestimate safety risks, and finetuned LLM detectors require curated data and extensive training.These limitations motivate a detector that does not require dataset collection and further LLM finetuning.Source: §1 · p. 2
- GradSafe identifies safety-critical parameter slices where jailbreak prompts paired with compliance responses have similar loss-gradient patterns, unlike safe prompts.It analyzes gradients rather than textual features of prompts or responses.Source: §1 · p. 2
- GradSafe analyzes gradients from a small set of reference safe and jailbreak prompts paired with “Sure,” then compares test-prompt gradients with an unsafe reference.GradSafe-Zero thresholds average cosine similarities, while GradSafe-Adapt uses them as features for logistic regression.Source: §1 · p. 2
- On ToxicChat and XSTest, Llama-2-based GradSafe-Zero without further training surpasses Llama Guard and leading online moderation APIs, while GradSafe-Adapt improves on ToxicChat.The introduction also reports that GradSafe-Zero outperforms state-of-the-art detection models and online moderation APIs across the two benchmarks.Source: §1 · p. 2
2 Related Work
Unsafe prompts can enable misuse and malicious finetuning, so detection is relevant both to deployed LLM applications and to customization services. Prior approaches include moderation APIs, benchmarks, and finetuned LLM detectors.
2.1 Threats of Unsafe Prompts to LLM
- Unsafe prompts can elicit harmful outputs despite safety alignment, and detecting them can help screen LLM applications against misuse.The paper also describes malicious finetuning on even a small number of unsafe prompts paired with compliant responses as a threat to alignment.Source: §2.1 · p. 3
- Screening unsafe prompts from user-provided finetuning data can help protect customization services from malicious finetuning.The paper reports that existing online finetuning services fail to detect such prompts effectively.Source: §2.1 · p. 3
2.2 Unsafe Prompt Detection
- Online moderation APIs are typically based on models trained with large datasets and processes such as data collection, labeling, training, and active learning.The cited examples include OpenAI Moderation API, Azure API, and Perspective API.Source: §2.2 · p. 3
- ToxicChat focuses on real user queries, XSTest tests exaggerated safety, and Llama Guard detects LLM input-output unsafety by finetuning Llama-2 on a collected dataset.The paper distinguishes GradSafe as a method that analyzes safety-critical gradients without further LLM finetuning.Source: §2.2 · p. 3
3 GradSafe
GradSafe distinguishes unsafe from safe prompts through gradient patterns on selected parameter slices. Its zero-shot variant thresholds cosine similarities, while its adaptation variant learns a logistic-regression detector from available labeled data.
3.1 Overview
- GradSafe detects unsafe prompts by comparing gradients from a prompt paired with a compliance response against an unsafe-gradient reference on safety-critical parameters.The framework has zero-shot and adapted variants.Source: §3.1 · p. 3
3.2 Identifying Safety-Critical Parameters
- Safety-critical slices are selected where unsafe-prompt gradients are similar to each other but differ from safe-prompt gradients; the corresponding unsafe reference gradients are stored.The filtering process averages unsafe gradients, computes slice-to-reference cosine similarities for safe and unsafe samples, then marks slices with sufficiently large similarity gaps; Figure 3 illustrates the phases and Table 1 reports marked-slice percentages by threshold.Source: §3.2 · pp. 3–4, Figure 3 · p. 4, Table 1 · p. 4
3.3 GradSafe-Zero
- GradSafe-Zero averages cosine similarities across safety-critical slices and classifies a prompt as unsafe when the score exceeds a predetermined threshold.The prompt is paired with “Sure,” and its loss gradients are computed with respect to the selected parameters.Source: §3.3 · p. 5
3.4 GradSafe-Adapt
- GradSafe-Adapt trains logistic regression on cosine similarities and labels from available training data to adjust detection to a target domain.At inference, the extracted similarities are fed to the classifier for detection.Source: §3.4 · p. 5
4 Experiment
Experiments evaluate GradSafe on ToxicChat and XSTest, comparing zero-shot detection, data-efficient adaptation, and design ablations. GradSafe-Zero leads the Llama-2-based detectors, while adaptation and ablations examine data needs and the role of safety-critical parameters and paired responses.
4.1.1 Dataset; 4.1.2 Evaluation Metrics; 4.1.3 Baselines
- The experiments use ToxicChat and XSTest, compare AUPRC plus precision, recall, and F1, and include online APIs, zero-shot LLMs, and finetuned Llama Guard as baselines.ToxicChat provides official train and test sets; XSTest has 250 safe prompts and 200 crafted unsafe prompts, with no training set.Source: §4.1.2 · p. 5, §4.1.1 · p. 5, §4.1.3 · p. 5
4.1.4 Settings for GradSafe
- GradSafe uses Llama-2 (Llama-2-7b-chat-hf) and pairs detection prompts with the response “Sure” to calculate gradients; GradSafe-Zero uses a 0.25 detection threshold when computing precision, recall, and F1 on both benchmarks.For safety-critical-parameter identification, GradSafe uses a gap threshold of 1.Source: §4.1.4 · p. 6
4.2 Overall Results
- GradSafe-Zero achieves the highest performance among the three Llama-2-based detectors, while GPT-4 performs better on XSTest and general moderation APIs are not effective enough on prompt-safety benchmarks.GPT-4 is especially strong on XSTest's less complex short prompts; the authors attribute the XSTest gap to using Llama-2 as the base model and lacking access to GPT-4 gradients.Source: §4.2 · pp. 6–7
4.3 Adaptability Study
- On ToxicChat, GradSafe-Adapt using 20% of training data achieves performance similar to Llama Guard fine-tuned with 100%.GradSafe-Adapt trains logistic regression on cosine-similarity features and labels rather than finetuning the LLM.Source: §4.3 · p. 7
4.4.1 Safety-Critical Parameters
- Safety-critical parameter identification makes detection more effective and reduces storage and computation; in adaptation, logistic-regression training can partly select important parameters.Without identification, general gradient similarities are noisier and less effective; the performance disparity is smaller for the adaptation variant.Source: §4.4.1 · p. 8
4.4.2 Reference Safe/Unsafe Prompts
- On XSTest, increasing the number of unsafe reference prompts improves AUPRC and reduces variation, while varying safe reference prompts has less pronounced effects.The authors relate the difference to unsafe references informing the safety-critical parameters and unsafe gradient reference more directly.Source: §4.4.2 · p. 8
4.4.3 Paired Response
- On XSTest, compliance and rejection paired responses yield favorable detection performance, whereas a neutral response does not.The tested responses are “Sure,” “I’m sorry,” and “I,” respectively.Source: §4.4.3 · p. 9
5 Discussion and Limitation
The authors frame GradSafe as a proof of concept and identify its limited exploration of base models and lack of fine-grained prompt classification as boundaries for further work.
5 Discussion and Limitation
- GradSafe is presented as a proof of concept with substantial room for improvement and future exploration.The discussion specifically notes that the method is not thoroughly evaluated across other LLMs.Source: §5 · p. 9
- The authors hypothesize that GradSafe’s effectiveness may vary with the base LLM and its safety alignment.They suggest the observed safety-critical gradient patterns may reflect gradients that disrupt alignment.Source: §5 · p. 9
- GradSafe assesses prompt safety comprehensively but does not classify prompts into specific unsafe categories.Fine-grained classification is deferred to future work; Table 7 is described as comparing base LLMs on XSTest using AUPRC.Source: §5 · p. 9, Table 7 · p. 9
6 Conclusion
The conclusion presents GradSafe as a safety-critical-parameter approach to detecting unsafe prompts and reports that it can outperform finetuned models without additional training on the original LLM.
6 Conclusion
- GradSafe examines LLM safety-critical parameters to detect unsafe prompts and is reported to outperform finetuned models without additional training on the original LLM.Source: §6 · p. 9
7 Ethical Impact
The ethical-impact discussion frames GradSafe as a safeguard against misuse and malicious finetuning while noting that open-sourcing the detector could support adaptive attacks.
7 Ethical Impact
- The work aims to safeguard LLMs from misuse, and its experiments reuse benchmark datasets rather than introducing new safety risks from unsafe samples.Source: §7 · p. 10
- The authors acknowledge that releasing the detection model could enable adaptive attacks based on its detection results.They also state that there are multiple ways to further improve the detector.Source: §7 · p. 10
- The authors state that the source code and software will be publicly available.Source: §7 · p. 10
A.1 Additional Experimental Details
The experiments used four Nvidia GeForce RTX 3090 GPUs for LLM inference and gradient calculations, and the reported result came from a single run under resource constraints and deterministic/stable results.
A.1 Additional Experimental Details
- Experiments requiring LLM inference and gradient calculations used four Nvidia GeForce RTX 3090 GPUs.Source: §A.1 · p. 11
- The reported result was generated in a single run, with resource constraints and deterministic/stable results cited as the reason.Source: §A.1 · p. 11
- The experiments used scikit-learn’s default setting to calculate all metrics.Source: §A.1 · p. 11
A.2 Reference Prompts in Identifying Safety-Critical Parameters
The appendix gives two reference safe prompts and two reference unsafe prompts used to identify safety-critical parameters, and states that only these four prompts are used. It also provides examples of each prompt type.
A.2 Reference Prompts in Identifying Safety-Critical Parameters
- Only four reference prompts—two safe and two unsafe—are used to identify safety-critical parameters.The safe examples are a cake-making request and a pet-story request; the unsafe examples are a fake-news request and a prompt requesting an unrestrained harmful response.Source: §A.2 · pp. 11–12
- The unsafe examples include a request to write fake news on China and a prompt demanding an unrestrained harmful response.The fake-news prompt is attributed to Xie et al. (2023), while the extended prompt is attributed to jailbreakchat.com.Source: §A.2 · p. 11
- The safe reference prompts ask for cake-making instructions and a story about pet animals.Source: §A.2 · p. 12
A.3 Prompts Pool for Ablation Study
The ablation prompt pool lists ten unsafe prompts and ten safe prompts. The examples range from requests for misinformation and harmful instructions to benign creative, instructional, and everyday tasks.
A.3 Prompts Pool for Ablation Study
- The unsafe ablation pool contains ten prompts, including requests for misinformation, harmful instructions, and deceptive claims.Examples include fake news, bomb-making instructions, and claims about a bank's impending bankruptcy.Source: §A.3 · p. 12
- Both pools include a storyteller-style instruction template, paired with a fake-news request in the unsafe pool and a vacation story prompt in the safe pool.Source: §A.3 · p. 12
- The safe ablation pool contains ten prompts spanning benign creative writing, recipes, how-to guides, and practical explanations.Examples include a cake recipe request, a pet story, dog-training guidance, and budgeting steps.Source: §A.3 · p. 12