PaperlayerView exact revision on arXiv

Source-linked AI summary

Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks

Chen Xiong, Xiangyu Qi, Pin-Yu Chen, Tsung-Yi Ho

arXiv:2405.20099v2cs.CR

TL;DR

Jailbreak attacks can circumvent LLM alignment, while prompt-based defenses can trade utility for defense or leave attack coverage limited. DPP optimizes an interpretable suffix using refusal and helpfulness objectives; it reports low ASR with minimal utility impact across two models and multiple attacks.

  • Problem

    Jailbreak attacks can elicit harmful outputs despite alignment, while prompt-based defenses face a trade-off between mitigating attacks and preserving utility.

    Source: §1 · pp. 1–2

  • Method

    DPP uses a Hierarchical Genetic Algorithm to optimize a readable suffix prompt by balancing refusal on harmful inputs with helpfulness on benign ones.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    DPP reports 3.8% ASR on Llama-2-7B-Chat and 2.0% ASR on Mistral-7B-Instruct-v0.2, with minimal utility loss.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Tests on two models and seven jailbreak strategies support DPP's potential as a practical prompt-based defense that reduces ASR with minimal impact on model performance.

    Source: §5 · p. 9

    DPP training's iterative optimization and evaluation may limit scalability as datasets or model deployments grow, although deployment after training is described as efficient.

    Source: Limitation · p. 9

Abstract

from arXiv · show

Safety, security, and compliance are essential requirements when aligning large language models (LLMs). However, many seemingly aligned LLMs are soon shown to be susceptible to jailbreak attacks. These attacks aim to circumvent the models' safety guardrails and security mechanisms by introducing jailbreak prompts into malicious queries. In response to these challenges, this paper introduces Defensive Prompt Patch (DPP), a novel prompt-based defense mechanism specifically designed to protect LLMs against such sophisticated jailbreak strategies. Unlike previous approaches, which have often compromised the utility of the model for the sake of safety, DPP is designed to achieve a minimal Attack Success Rate (ASR) while preserving the high utility of LLMs. Our method uses strategically designed interpretable suffix prompts that effectively thwart a wide range of standard and adaptive jailbreak techniques. Empirical results conducted on LLAMA-2-7B-Chat and Mistral-7B-Instruct-v0.2 models demonstrate the robustness and adaptability of DPP, showing significant reductions in ASR with negligible impact on utility. Our approach not only outperforms existing defense strategies in balancing safety and functionality, but also provides a scalable and interpretable solution applicable to various LLM platforms.

1 Introduction

Jailbreak attacks can bypass alignment, while existing prompt-based defenses struggle to balance attack mitigation with model utility. DPP uses an optimized, readable suffix to target both goals, reporting low ASR and minimal utility loss.

1 Introduction

  • Existing prompt-based defenses exhibit different safety-utility trade-offs: Goal Prioritization compromises utility, RPO has limited defense coverage, and Self-Reminder performs poorly on Llama-2-7B-Chat.The paper attributes Self-Reminder's result on that model to deficiencies in its search algorithm.Source: §1 · pp. 1–2
  • DPP achieved 3.8% ASR on Llama-2-7B-Chat and 2.0% ASR on Mistral-7B-Instruct-v0.2 while maintaining minimal utility loss.The paper reports these model-specific outcomes as evidence of DPP's defense-utility balance.Source: §1 · p. 2
  • DPP is a human-readable prompt defense trained with a Hierarchical Genetic Algorithm that refines a suffix for every input query without model retraining.The search alternates sentence-level word substitutions and paragraph-level crossovers, retaining patches that raise refusal likelihood on harmful inputs while preserving helpfulness on benign ones.Source: §1 · p. 2
  • Across adaptive and unforeseen jailbreak strategies, DPP consistently achieved the lowest average attack success rate.The authors report this result across multiple scenarios.Source: §1 · p. 2
  • The authors report clearer DPP prompts than existing prompt-based defenses and better defense and utility when the patch is a suffix rather than a prefix.The suffix-prefix comparison is an ablation on Llama-2-7B-Chat.Source: §1 · p. 2

2 Related Work

The related work surveys jailbreak strategies and prompt-based defenses, then frames the need for defenses that maintain both safety and utility. Its overview figure depicts DPP training, deployment, and reported utility-ASR comparisons.

2 Related Work

  • Jailbreak strategies include attacker-LLM-generated prompts, in-context learning, generation-configuration manipulation, gradient-based inputs, genetic prompt refinement, and Base64 encoding.PAIR and TAP use a secondary attacker LLM; AutoDAN refines prompts with genetic algorithms.Source: §2 · p. 2
  • Prompt-based defenses modify prompts or objectives to reinforce safety, including Self-Reminder, RPO, Goal Prioritization, and Default System Prompts.These approaches are presented as responses to jailbreak attacks.Source: §2 · p. 2
  • Table 1 is presented as a comparison of defense methods against jailbreak attacks on LLMs.The related-work text says the comparison elucidates trade-offs among prompt-based defense strategies.Source: Table 1 · p. 3
  • Figure 1 depicts DPP training from refusal and helpful datasets, suffix deployment on each query, and utility-win-rate versus ASR plots for two models.The figure caption identifies Llama-2-7B-Chat and Mistral-7B-Instruct-v0.2 in the trade-off plots.Source: Figure 1 · p. 3

3 Methodology

The methodology defines defense and utility objectives for a prompt patch, combines their scores, and uses hierarchical genetic search to optimize a suffix. Candidate patches are iteratively refined and selected for their combined performance.

3.1 Preliminaries

  • DPP appends a defensive patch to a malicious query to elicit a refusal, while utility degradation measures how patched benign-query responses deviate from their unpatched responses.The method evaluates both defense against malicious inputs and impact on benign inputs.Source: §3.1 · pp. 3–4

3.2 Score Evaluation

  • The total training score combines refusal and helpfulness scores, weighted by α and β to balance the two objectives.The refusal score concerns jailbreak queries and refusal responses; the helpfulness score concerns benign queries and standard responses.Source: §3.2 · p. 4

3.3 DPP Training Algorithm

  • DPP training uses hierarchical genetic search, alternating sentence-level word substitution with paragraph-level sentence swaps and mutations.Candidates are scored, selected, and refined through the two operations.Source: §3.3 · pp. 4–5
  • The search evaluates a rewritten prototype using sampled refusal and helpful data, retains candidate patches, and returns the patch with the highest total score.The optimization iterates across multiple refusal and helpful data pairs to seek a more generalizable patch.Source: §3.3 · pp. 4–5

4 Experiments

Experiments assess DPP’s robustness to standard and adaptive jailbreaks, generalization across models and unseen queries, prompt clarity, and the contribution of design choices. Across the reported evaluations, DPP achieves low attack success rates, with utility trade-offs varying by model.

4.1 Experimental Setup

  • Experiments assess jailbreak defenses using 100 AdvBench harmful questions and 100 Alpaca benign questions across Llama-2-7B-Chat and Mistral-7B-Instruct-v0.2, measuring attack success with ASR and utility degradation with Win-Rate.Baselines include Self-Reminder and Goal Prioritization, plus RPO for Llama-2-7B-Chat and Plain (Default) System Prompt for Mistral-7B-Instruct-v0.2.Source: §4.1 · pp. 5–6

4.2 Robustness against Non-adaptive and Adaptive Attacks

  • 3.80% Average ASR and 82.98% Win-Rate on Llama-2-7B-Chat lead non-adaptive results; adaptive attacks yield 13.0% Average ASR, best among baselines.The adaptive result outperforms the second-best method by more than 4%.Source: §4.2 · pp. 6–7, Table 2 · p. 7, Table 3 · p. 7

4.3 Generalization of DPP

  • More than 20% lower average ASR than Goal Prioritization on Mistral non-adaptive attacks, but higher utility degradation than Self-Reminder and System Prompt.The reported defense-performance gap exceeds 46% in average ASR.Source: §4.3 · p. 7, Table 4 · p. 8
  • DPP has the lowest average ASR on Mistral adaptive attacks and achieves 5.4% average ASR with Llama-Guard judging.The 5.4% result is reported for the Llama-Guard evaluation in Appendix L.Source: §4.3 · p. 8, Table 5 · p. 8
  • On 100 independent AdvBench harmful queries, DPP reaches 7.5% Average ASR on Llama-2-7B-Chat and 39.4% on Mistral-7B-Instruct-v0.2.These queries were independent of the Adversarial Dataset.Source: §4.3 · p. 8

4.4 Clarity of DPP

  • Both model-specific DPPs are more fluent than RPO; the Mistral patch explicitly refuses “defective components,” while the Llama patch prompts a thorough response.The paper attributes this difference to Mistral’s comparatively weaker alignment.Source: §4.4 · p. 8

4.5 Ablation Study

  • Ablations report that removing either helpfulness or defense reduces ASR or Win-Rate; additional tests favor suffixes, varied initialization, HGA over RLPrompt, and sentence-level substitutions.The study includes No Defense and No Helpful settings.Source: §4.5 · pp. 8–9

5 Conclusion

The paper presents DPP as a scalable, practical prompt-based safeguard evaluated across two models and seven jailbreak strategies. Results support lower attack success with limited utility impact, while future work targets utility degradation on less-aligned models.

5 Conclusion

  • Across Llama-2-7B-Chat and Mistral-7B-Instruct-v0.2 and seven jailbreak strategies, DPP reduces attack success while preserving utility, including on the less-aligned model.The paper describes the approach as scalable, practical, and interpretable.Source: §5 · p. 9

Limitation

DPP has training-cost and deployment-scope constraints, while its human-readable design is intended to support inspection, model-agnostic use, and nonexpert refinement. The authors also identify specific weaknesses in competing defenses.

Limitation

  • Iterative DPP optimization may constrain scaling to larger datasets or deployments, although trained patches have high inference efficiency when appended to queries.The training algorithm is computationally intensive; deployment requires only attaching the trained DPP to the user query.Source: Limitation · p. 9
  • Using GPT-4 to revise prototype prompts makes DPP training costly; replacing it with an open-source LLM is proposed as future work.The authors describe access to the OpenAI platform as considerably costly for this training process.Source: Limitation · p. 9
  • The authors report that Self-Reminder performs poorly on Llama-2-7B-Chat, RPO incurs high training cost, and Goal Prioritization has attack-coverage, inference-time, and utility drawbacks.Goal Prioritization is described as less effective against AutoDAN, TAP, and PAIR; utility degradation is observed on both Llama-2-7B-Chat and Mistral-7B-Instruct-v0.2.Source: Limitation · p. 9
  • DPP’s open-weight-model use case targets providers, since local users can remove system prompts and leave models vulnerable to jailbreaks.The authors state that DPP cannot protect the actual safety of an open-weight model when malicious local users can remove its prompts.Source: Limitation · p. 9

A Motivation for Human-Readable Defense Prompts

  • Human-readable patches are intended to aid developer and auditor verification, support model-agnostic deployment, and remain inspectable or refinable by nonexperts.The authors cite clarity and stability, generalization across LLMs, and a human-centric design philosophy as motivations.Source: A Motivation for Human-Readable… · p. 12

B Ablation Studies on DPP algorithm

The ablations examine DPP’s objective, patch placement, prototype initialization, optimizer, and synonym substitution. Their reported comparisons favor suffix patches, HGA over RLPrompt, and the full algorithm with synonym substitution.

B.1 Ablation study on different objective functions

  • The objective-function ablation reports ASR and Win-Rate results for No Defense and No Helpful settings.Table 6 is described as masking out different objective functions; supplied passages do not state the cell values.Source: §B.1 · p. 12, Table 6 · p. 12

B.2 Ablation study on different patching format

  • Suffix DPP exceeded Prefix DPP by 3% in average Win-Rate; its ASR was 7% lower under non-adaptive GCG and differed by 42% under adaptive GCG.The authors recommend suffixes as the default format, particularly given Prefix DPP’s weaker performance under adaptive settings.Source: §B.2 · p. 12, Table 7 · p. 13

B.3 Ablation study on different prototype selections

  • With two Self-Reminder prototype variants, DPP improved both defense and utility in non-adaptive and adaptive GCG settings on Llama-2-7B-Chat.The reported comparisons are in Tables 9 and 10.Source: §B.3 · p. 13, Table 9 · p. 13, Table 10

B.4 Ablation study on different solver for our DPP optimization formulation

  • RLPrompt achieved a GCG ASR of 0.15 versus HGA’s 0.04, and a Win-Rate of 47.89 versus HGA’s 82.98 on Llama-2-7B-Chat.The comparison uses the same objective function, defined as Eq. 6.Source: §B.4 · p. 13, Table 11 · p. 13

B.5 Ablation study on the necessity of sentence-level synonym substitution

  • The full DPP algorithm outperformed the version without sentence-level synonym substitution in jailbreak defense and utility trade-off.The study retrained the Llama-2-7B-Chat pipeline with synonym substitution disabled while keeping the other components unchanged.Source: §B.5 · p. 14, Table 12 · p. 14

C Implementation Details

This section details DPP’s implementation and evaluation, covering model-specific settings and optimization procedures, example suffixes and attack setups, evaluation metrics, and how DPP is incorporated into adaptive attacks.

C Implementation Details

  • On Llama-2-7B-Chat, α = 1 and β = 10; on Mistral-7B-Instruct-v0.2, α = 10 and β = 1, with training on one 80GB A800 costing approximately $75.The reported time-cost analysis uses one training instance per epoch and 100 epochs.Source: C Implementation Details · p. 14

D DPP Supplementary Functions

  • DPP supplements generate K meaning- and length-preserving GPT-4 rewrites, rank words by average Eq. 4 and Eq. 5 scores, and use sentence swapping and LLM revision to transform prompts.The training procedure initializes refusal and helpful datasets, iteratively optimizes the DPP set, and selects its best DPP.Source: D DPP Supplementary Functions · pp. 15–16

E DPP Suffix

  • The appendix shows Llama-2-7B-Chat suffix examples that prompt helpful answers or warn against harmful content, plus a Mistral suffix that begins with “I’m sorry” for defective-component queries.It presents three Llama suffix examples and one Mistral suffix example.Source: E DPP Suffix · p. 16

F Jailbreak Prompt Generations

  • The evaluations cover three jailbreaking attack types: uninterpretable, interpretable, and generation-based; GCG and Base64 are classified as uninterpretable, while AutoDAN, PAIR, and TAP are classified as interpretable attacks.Source: F Jailbreak Prompt Generations · p. 17

G Performance Investigation for RPO

  • For Starling-7B, the authors report that RPO provides insufficient defense on less-aligned models and therefore do not consider it a sufficient baseline for Mistral.The accompanying table is scoped to RPO on Starling-7B under non-adaptive GCG and AutoDAN attacks.Source: G Performance Investigation for RPO · p. 18, Table 14 · p. 18

H Prompts in Defense Baselines

  • The defense-baseline prompts include model-specific self-reminders and instructions to prioritize safety, refuse potentially harmful requests, and answer safe queries helpfully.The examples include a safe study-advice query and a harmful DOS-instructions query.Source: H Prompts in Defense Baselines · pp. 18–19

I Attack Success Rate Evaluation Metrics

J Win-Rate Evaluation

  • Win-rate compares target-model responses with Davinci003, uses a 1000-token response limit, and removes system prompts for consistent evaluation.The authors report that longer responses generally yield higher win-rates and that default system prompts can limit answering capability; they also use GCG without a system prompt to match the original study more closely (Zou et al., 2023).Source: J Win-Rate Evaluation · p. 20

K Adaptive Attacks Setup

  • Adaptive GCG optimizes a jailbreak suffix with DPP appended, while adaptive ICA combines five in-context jailbreak demonstrations with DPP and the test query.For ICA, the demonstrations pair harmful user queries with their original harmful responses; the resulting prompt is intended to elicit refusal despite the jailbreak context.Source: K Adaptive Attacks Setup · p. 21
  • Adaptive AutoDAN, PAIR, and TAP expose their prompt-generation or optimization procedures to DPP while seeking jailbreak templates that still succeed.AutoDAN appends DPP during template optimization and uses synonym replacement, selection, crossover, and mutation; PAIR appends DPP to generated prompts, and TAP applies it to generated jailbreak prompts before evaluating responses.Source: K Adaptive Attacks Setup · p. 21
  • Catastrophic adaptive attack appends DPP to each malicious query and tests decoding-parameter combinations as a black-box attack.The tested settings vary temperature, top_p, and top_k, and responses are evaluated with the attack’s judge procedure.Source: K Adaptive Attacks Setup · p. 22

L Llama-Guard Judge Evaluation

The authors re-evaluate DPP and defense baselines with LlamaGuard on Llama-2-7B-Chat and Mistral-7B-Instruct-v0.2, using adversarial-dataset queries and a non-overlapping AdvBench test set. Across both query sets, DPP outperforms the other defense baselines under LLM judgment.

M Extension of Llama-2 Experiments

The Llama-2 extension tests three DPP suffix initializations, while the following section evaluates DPP on two additional jailbreak attacks under adaptive settings. The reported extension results describe average-ASR advantages over baselines, with some utility trade-offs across suffix variants.

M Extension of Llama-2 Experiments

N DPP performance on other Jailbreak Attacks

O Extension of Mistral Experiments

The Mistral extension compares DPP with Directed Representation Optimization (DRO) across six jailbreak attacks. DPP has lower Average ASR, while DRO has a better Win-Rate.

P JailbreakBench Chat Queries

JailbreakBench results show DPP reducing ASR on Mistral, while additional experiments relate model-specific DPP wording to alignment and jailbreak-query length.

P JailbreakBench Chat Queries

  • On Mistral-7B-Instruct-v0.2, DPP reduced JailbreakBench ASR from 41% to 1%, matching Goal Prioritization for best defense performance.The JBC experiments use data obtained from Vicuna-13B-v1.5 because Mistral-specific data were absent; the authors also attribute AutoDAN’s 0 ASR on Llama to a template mismatch.Source: P JailbreakBench Chat Queries · p. 24, Table 27 · p. 24

Q.1 Deeper Insights of DPPs

  • The authors hypothesize that Mistral’s greater vulnerability to heuristic jailbreaks motivates “defective components” in its DPP, unlike Llama’s.Their comparison of the models’ ASR on the same human-engineered JBC queries informs this hypothesis.Source: §Q.1 · pp. 24–25, Table 31 · p. 25
  • Successful jailbreak queries on Llama were longer than those on Mistral, motivating the authors’ “thorough” wording in Llama’s DPP.The authors connect this pattern to prompt dilution: longer queries might compromise Llama, so they suggest the model should read them carefully.Source: §Q.1 · p. 25, Table 32 · p. 25, Table 33 · p. 25

Q.2 Quantitative analysis of clarity between different defense mechanisms

The appendix compares DPP’s readability, defense and utility trade-offs, adaptive-attack performance, and broader impacts across several evaluations.

Q.2 Quantitative analysis of clarity between different defense mechanisms

  • DPP had lower perplexity than RPO and Self-Reminder, indicating higher clarity, while remaining competitive with handcrafted prompts without human prompt-design intervention.The perplexity comparison used GPT-4.Source: §Q.2 · p. 25, Table 34 · p. 25

R DPP performance on advanced jailbreak attacks on Llama-2-7B-Chat

S Trade-off Plots

  • Trade-off plots show DPP outperforming baselines on both utility and defense for Llama, while on Mistral it defended better but degraded utility more than Self-Reminder and System Prompt.The comparison covers adaptive and non-adaptive settings.Source: Figure 2 · p. 26, Figure 4 · p. 26, Figure 3 · p. 26

T Ignorance Adaptive Attack

  • Under the ignorance-adaptive attack, defenses performed similarly on Llama-2-7B-Chat, while DPP outperformed all baselines on Mistral-7B-Instruct-v0.2.The authors attribute Llama’s result to its existing alignment making the malicious queries ineffective.Source: T Ignorance Adaptive Attack · p. 26, Table 36 · p. 27

U Unforeseen Jailbreak Queries on Llama-2-7B-Chat and Mistral-7B-Instruct-v0.2

V Broader Impacts

  • The authors note that DPP may produce false positives that hinder legitimate use, and propose monitoring, iterative improvement, and transparent vulnerability reporting.They state that they do not foresee negative impacts from the work.Source: V Broader Impacts · p. 27

W.1 Training DPP on Vicuna

Appendix evaluations report DPP results on Vicuna and Llama-3, test transfer across models, introduce Min Over Prompt, and point to a repository.

W.1 Training DPP on Vicuna

  • On Vicuna-13B-v1.5, DPP had the best defense performance by average ASR under adaptive evaluation against four jailbreak attacks.RPO was omitted because obtaining its defense suffix requires time-consuming training.Source: §W.1 · pp. 27–28, Table 39 · p. 28

W.2 DPP transferability performance on Vicuna

  • A DPP trained on Mistral and applied directly to Vicuna outperformed the other defense baselines in defense ability.The evaluation used an adaptive setting.Source: Table 40 · p. 28, §W.2 · p. 28

X DPP performance on Llama-3-8B-Instruct

Y Min Over Prompt Evaluation

Z Repository

  • The authors released an anonymous repository containing trained DPPs for Llama-2-7B-Chat and Mistral-7B-Instruct-v0.2.The repository link is provided in the paper.Source: Z Repository · p. 29

Related papers

Loading 2405.20099v2…