PaperlayerView exact revision on arXiv

Source-linked AI summary

SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner

Xunguang Wang, Daoyuan Wu, Zhenlan Ji, Zongjie Li, Pingchuan Ma, Shuai Wang, Yingjiu Li, Yang Liu, Ning Liu, Juergen Rahmel

arXiv:2406.05498v3cs.CRcs.AI

TL;DR

Jailbreaks bypass LLM safety alignment, and practical defenses must address varied attacks with negligible delay while supporting open- and closed-source models. SELFDEFEND pairs a shadow LLM detector with a target LLM, and it outperforms other defenses in most tested scenarios while keeping reported delays low.

  • Problem

    Jailbreaks can bypass LLM safety alignment, while practical defenses must cover varied attacks, incur negligible delay, and work with open- and closed-source models.

    Source: §1 · pp. 1–2

  • Method

    SELFDEFEND runs a shadow LLM detector alongside the target LLM, using checkpoint-based access control to coordinate detection and answering.

    Source: §1 · p. 2, §4 · p. 6

  • Results

    SELFDEFEND outperforms all other defenses in 55 of 60 tested attack scenarios and reaches the defense level of GPT-4-based SELFDEFEND.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluation characterizes SELFDEFEND as lightweight and effective across a wide spectrum of jailbreak attacks while rarely undermining normal queries.

    Source: §9 · p. 14

    The threat model covers text-based attacks and excludes multimodal jailbreaks; it assumes the success classifier accurately judges harmful outputs.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Jailbreaking is an emerging adversarial attack that bypasses the safety alignment deployed in off-the-shelf large language models (LLMs) and has evolved into multiple categories: human-based, optimization-based, generation-based, and the recent indirect and multilingual jailbreaks. However, delivering a practical jailbreak defense is challenging because it needs to not only handle all the above jailbreak attacks but also incur negligible delays to user prompts, as well as be compatible with both open-source and closed-source LLMs. Inspired by how the traditional security concept of shadow stacks defends against memory overflow attacks, this paper introduces a generic LLM jailbreak defense framework called SelfDefend, which establishes a shadow LLM as a defense instance (in detection state) to concurrently protect the target LLM instance (in normal answering state) in the normal stack and collaborate with it for checkpoint-based access control. The effectiveness of SelfDefend builds upon our observation that existing LLMs can identify harmful prompts or intentions in user queries, which we empirically validate using mainstream GPT-3.5/4 models against major jailbreak attacks. To further improve the defense's robustness and minimize costs, we employ a data distillation approach to tune dedicated open-source defense models. When deployed to protect GPT-3.5/4, Claude, Llama-2-7b/13b, and Mistral, these models outperform seven state-of-the-art defenses and match the performance of GPT-4-based SelfDefend, with significantly lower extra delays. Further experiments show that the tuned models are robust to adaptive jailbreaks and prompt injections.

1 Introduction

Jailbreaks can bypass LLM safety alignment, while practical defenses must cover varied attacks with negligible delay and work across open- and closed-source models. SELFDEFEND pairs a shadow LLM detector with a target LLM, and its reported evaluations show broad defense effectiveness and low delay.

1 Introduction

  • The paper frames a practical defense as needing to cover varied jailbreak attacks, add negligible prompt delay, and support both open- and closed-source LLMs.Source: §1 · pp. 1–2
  • SELFDEFEND uses a shadow LLM for jailbreak detection alongside a target LLM, combining detection with the target’s safety alignment through dual-layer protection.The target answers normally while the shadow instance checks the query and triggers checkpoint-based access control.Source: §1 · p. 2
  • GPT-3.5- and GPT-4-based SELFDEFEND lower average ASR to 0.236 and 0.050, respectively, with normal-prompt pass rates almost unaffected for GPT-3.5.For GPT-4, the normal-prompt pass rate decreases by 2.77%.Source: §1 · p. 2
  • The authors distill GPT-4-generated data from 38,961 harmful and harmless prompts, then LoRA-fine-tune Llama-2-7b as an open-source defense model.The approach is intended to provide robust, low-cost, self-contained defense.Source: §1 · p. 2
  • SELFDEFEND consistently outperforms all other defenses in 55 of 60 tested attack scenarios and reaches the defense level of GPT-4-based SELFDEFEND.Source: §1 · p. 2
  • For normal prompts, average extra delay is 0–0.01 seconds; for attacks, delay is below 0.1 seconds except for DAN and LLM-Fuzzer.The maximum attack delay falls from 1.56 seconds with GPT-4-based SELFDEFEND to 0.39 seconds with tuned SELFDEFEND.Source: §1 · p. 2

2 Background

The threat model considers text-based jailbreaks against an LLM accessible through its interface, with success judged against a harmful goal. The background groups attacks into human-based, optimization-based, generation-based, indirect, and multilingual approaches.

2.1 Threat Model

  • The threat model excludes multimodal jailbreaks and assumes an adversary can submit arbitrary prompts through the target LLM’s interface.The adversary seeks a response judged successful against a predefined harmful goal; the setup assumes the classifier judges success accurately.Source: §2.1 · p. 3

2.2 Jailbreak Attacks

  • Human-based attacks manually craft prompts, while optimization-based attacks iteratively search or update adversarial prompts and may produce transferable suffixes.The section describes GCG as optimizing suffixes and also discusses human-readable optimization approaches.Source: §2.2 · p. 3
  • Generation-based attacks use language models to create jailbreak prompts, including approaches that adapt prompts using target-model responses or mutate templates.Source: §2.2 · p. 3
  • Indirect jailbreaks conceal malicious intent through substitutions, decomposed prompts, or clues, while multilingual attacks translate or obfuscate harmful prompts.Source: §2.2 · pp. 3–4

3 Objectives and Related Work

The paper sets four objectives for jailbreak defenses: broad attack coverage, negligible delay, explanations for potential attacks, and compatibility across open- and closed-source LLMs. Its review contrasts plugin-based and model-based defenses against these objectives.

3.1 Design Objectives

  • The proposed objectives are broad jailbreak coverage, negligible delay to normal prompts, explanations for potential jailbreaks, and compatibility with open- and closed-source LLMs.Explanations are described as valuable but not mandatory.Source: §3.1 · p. 4

3.2 Analysis of Existing Defenses

  • Most reviewed defenses target multiple attack types but often omit advanced indirect attacks; some are designed specifically for optimization-based attacks.Source: §3.2 · p. 4
  • Most plugin-based defenses add delay through extra prompt analyses or checks of target-model states or responses, whereas most model-based methods avoid extra prompt delay.Source: §3.2 · p. 4
  • More than half of plugin-based defenses can provide explanations, while most model-based defenses cannot because they rely on internal mechanism tuning.Source: §3.2 · p. 4
  • Most plugin-based defenses work with open- and closed-source LLMs, while most model-based defenses require white-box access to the target.Source: §3.2 · p. 4

4 The SELFDEFEND Framework

SELFDEFEND deploys a shadow LLM alongside a target LLM so detection can run concurrently with answering. Its checkpoint-based workflow releases the target’s response when the detector signals no issue and otherwise returns a refusal.

4 The SELFDEFEND Framework

  • Table 1 compares existing jailbreak defenses under the four objectives and distinguishes plugin-based defenses from model-based defenses by row group.Source: Table 1 · p. 5
  • SELFDEFEND pairs a target LLM in the normal stack with a shadow defense LLM, combining the target’s safety alignment with dedicated jailbreak detection.The defense LLM can use the same model as the target, though the authors recommend a robust, low-cost dedicated detector.Source: §4 · pp. 4, 6
  • The target processes and caches its answer while the shadow LLM detects harmful content or intent; a “No” signal releases the answer, otherwise the system refuses.The refusal can identify the recognized harmful portion.Source: §4 · p. 6
  • Because the defense LLM need not modify or inspect the target’s internals, SELFDEFEND can protect open- and closed-source models while providing explanations for detected attacks.The authors also state that the detector’s short “No” output makes normal-prompt delay negligible.Source: §4 · p. 6

5 An Empirical Measurement

This empirical study tests whether GPT-3.5/4 can support SELFDEFEND and evaluates its detection prompts across representative jailbreak benchmarks. It reports prompt-dependent defense improvements, near-baseline normal-prompt performance, and low added delay.

5.1 Initializing SELFDEFEND for GPT-3.5/4 and Designing Two Detection Prompts

  • Pdirect identifies potentially harmful portions directly; Pintent summarizes the query’s true intent, extracts any safety-violating content, and returns “No” for safe queries.Pintent includes two contrasting examples.Source: §5.1 · p. 7

5.2 Datasets and Attack Setup

  • Four datasets supply prompts across five jailbreak categories and normal queries; attack success uses refusal-keyword ASR, while multilingual MultiJail uses GPT-4-judged unsafe rate.One hundred JailbreakBench harmful instructions drive optimization-, generation-, and indirect-attack testing.Source: §5.2 · p. 7

5.3 Measurement Results

  • GPT-3.5’s Pintent outperformed Pdirect (reduction factors 97.26% to 8.97%), while GPT-4’s Pdirect was strongest (69.69% to 100.00%).Table 3 summarizes the ASR results across jailbreak attacks and normal prompts.Source: Table 3 · p. 8, §5.3 · p. 8
  • Pdirect reduced normal-prompt ASR by 2% (from 0.977 to 0.957) for GPT-3.5 and 2.7% (from 0.973 to 0.946) for GPT-4; Pintent hardly reduced it.Source: Table 3 · p. 8, §5.3 · p. 8
  • GPT-3.5 baseline ASR ranged from 0.256 to 0.980 (average 0.655), while GPT-4 was generally lower but reached 0.74 on DrAttack and 0.900 on Puzzler.Source: §5.3 · p. 8
  • Over 95% of normal-prompt cases incurred zero extra delay across three configurations, except GPT-3.5 with Pintent, where around 80% had none.GPT-3.5’s average extra delay was under 0.1 seconds across all 10 attack scenarios.Source: Figure 3 · p. 8, §5.3 · p. 8

6 Tuning a Dedicated Defense Model

The section transfers GPT-4-based SELFDEFEND behavior to an open-source defense model through distilled examples and LoRA fine-tuning. The resulting models are trained on direct-prompt and intent-prompt datasets built from single-turn harmful and harmless queries.

6 Tuning a Dedicated Defense Model

  • GPT-4’s cost and closed-source privacy concerns motivate tuning an open-source model for low-cost, self-contained SELFDEFEND protection.The preceding GPT-4-based framework lowered average ASR to 0.063.Source: §6 · p. 9

6.1 Design Overview

  • GPT-4 labels harmful and harmless prompts wrapped with Pdirect or Pintent, producing separate datasets for fine-tuning an open-source defense model.Each defense prompt yields a separate dataset.Source: §6.1 · p. 9

6.2 Data Distillation

  • The Anthropic red-team dataset supplies 38,961 transcripts, reduced to English single-turn human prompts without assistant responses or later exchanges.The dataset predates and does not overlap with JailbreakHub, JailbreakBench, or MultiJail; it adds no multilingual capability.Source: §6.2 · p. 9
  • GPT-4 responses to each dataset prompt under Pdir and Pint form Ddir and Dint for tuning.The original human prompt is x and its corresponding GPT-4 response is y.Source: §6.2 · p. 9

6.3 LoRA Fine-Tuning

  • The authors adapt Llama-2-7b with LoRA, a parameter-efficient method chosen given limited GPU resources.Training uses r=8, α=32, one epoch, batch size 8, and an 80/20 train-validation split; inputs are defense-wrapped prompts and labels are distilled responses.Source: §6.3 · pp. 9–10
  • The direct-prompt objective is intended to improve detection of jailbreak parts, and the section also specifies an intent-prompt fine-tuning objective.LoRA trainable parameters are added to the pretrained model weights.Source: §6.3 · p. 9

7 Evaluation

The evaluation tests tuned SELFDEFEND models across target LLMs, jailbreak types, latency, explainability, adaptive attacks, and prompt injection. The tuned models achieve lower average ASRs than guard baselines in the reported comparisons, with low added delay and robustness in the tested attack settings.

7 Evaluation

  • The evaluation covers six target LLMs and compares defense effectiveness, delay, explainability, and robustness to adaptive attacks and prompt injection.Targets include GPT-3.5, GPT-4, Claude-3.5-sonnet, Llama-2-chat at 7B and 13B, and Mistral-7B-Instruct-v0.2.Source: §7 · p. 10, §7.1 · p. 10

7.1 Experimental Setup

  • ASR measures attack success, but its operational meaning varies by defense; the evaluation includes defenses, attack families, and normal prompts.The tested target set spans different model architectures and sizes.Source: §7.1 · p. 10

7.2 Defense Effectiveness

  • Tuned SELFDEFEND achieves satisfactory results across jailbreak types and is state-of-the-art in most cases, while comparator defenses show weaknesses on several attacks.Across human-based, optimization-based, generation-based, indirect, and multilingual attacks, the two prompt variants have different strengths: Pdirect does better on explicit-instruction attacks, while Pintent is more effective on indirect and multilingual attacks and has fewer false positives on normal prompts.Source: §7.2 · pp. 10–12, Table 4 · p. 11
  • 22.03% average ASR for Pdirect-tuned SELFDEFEND is below Llama Guard’s 66.08%, Llama Guard 2’s 41.10%, and Llama Guard 3’s 26.26% across attacks and targets.Pintent-tuned SELFDEFEND has 18.39% average ASR, outperforming those guards by 47.69%, 22.71%, and 7.87%, respectively; tuned models deliver state-of-the-art results in most cases.Source: §7.2 · pp. 10–11, Table 4 · p. 11

7.3 Extra Delay ∆d

  • On AlpacaEval, added delay is around 0 seconds for Pdirect and 0.008 seconds for Pintent; the maximum delay falls from GPT-4’s 1.56 seconds to 0.39 seconds.Except for DAN and LLM-Fuzzer, tuned-model delay is below 0.1 seconds across attack scenarios; the table compares mean delay across 11 tested inputs.Source: §7.3 · pp. 12–13, Table 6 · p. 13

7.4 Explainability

  • For successful defenses, identified harmful prompts are more similar to originals than harmful intentions on explicit attacks, but less similar on indirect and multilingual attacks.The comparison uses semantic similarity measured with ensemble CLIP-score.Source: Table 5 · p. 12, §7.4 · p. 13

7.5 Robustness to Adaptive Jailbreaks

  • Under separate adaptive attacks, Pintent-tuned SELFDEFEND averages 0.023 ASR versus Pdirect-tuned SELFDEFEND’s 0.193, and both tuned models outperform guard baselines in individual GCG and RLbreaker tests.Against GCG, their ASRs are 0.08 and 0.09, compared with 1.00 for Llama Guard and 0.99 for LLM SELF DEFENSE; against RLbreaker, both tuned models have 0.00 ASR while both baselines exceed 70%.Source: §7.5 · pp. 13–14, Table 8 · p. 14

7.6 Robustness to Prompt Injection

  • Against prompt injection tested with 100 JailbreakBench goals, Pintent-tuned SELFDEFEND has 0.17 ASR versus 0.21 for Pdirect-tuned SELFDEFEND.The authors interpret both values as low and the intent-tuned model as less affected by prompt injection.Source: §7.6 · p. 14

8 Discussion

SELFDEFEND pairs a target model’s normal answering state with a shadow model’s detection state, contrasting with guardrails that judge content without using the target model’s alignment. The discussion also describes how tuned models prioritize malicious intent and aim to preserve normal-prompt utility.

8 Discussion

  • SELFDEFEND uses one instance to answer normally and another to detect harmful content, collaborating with the target model’s safety alignment rather than judging content in isolation.The authors argue this could improve defense effectiveness over guardrails and report empirical tests of that comparison.Source: §8 · p. 14
  • Tuned models check direct malicious portions or indirect malicious intentions before making safety judgments, unlike Llama Guard’s judgment-first approach.The authors report greater effectiveness against jailbreaks and lower likelihood of being hacked than the Llama Guard series.Source: §8 · p. 14
  • SELFDEFEND is plug-in and self-contained, requires no target-model modifications, and shows similar ASRs to the target on normal AlpacaEval prompts.The authors expect normal user experience to remain unaffected.Source: §8 · p. 14

9 Conclusion

The conclusion presents SELFDEFEND as a shadow-LLM defense, strengthened through tuning, that mitigates diverse jailbreak attacks with low cost while rarely undermining normal queries.

9 Conclusion

  • SELFDEFEND uses a dual-layer defense in which a shadow LLM guards the target LLM, inspired by shadow stacks.The shadow LLM is the defense instance alongside the target model.Source: §9 · p. 14
  • A tuning-based approach further enhances the shadow LLM’s defense capability.Source: §9 · p. 14
  • Evaluation finds SELFDEFEND lightweight and effective against a wide spectrum of jailbreak attacks while rarely undermining normal queries.Source: §9 · p. 14

Ethics Considerations

The paper reports safeguards for conducting its jailbreak tests and releases its datasets, raw evaluation results, code, and artifact.

Ethics Considerations

  • The authors conducted ChatGPT tests on their own account and local server, spread experiments over three months, and incurred about $2,000 in API fees.They report complying with ChatGPT terms and Meta Llama agreements and protecting team members from harmful-content exposure.Source: Ethics Considerations · p. 15
  • The authors identify possible malicious use of the defense to craft better attacks and describe community engagement and readiness to take corrective action.They also state that their research complied with relevant law, including data-protection and privacy standards.Source: Ethics Considerations · p. 15

Open Science

  • The authors released their datasets, raw evaluation results, code, and artifact.Source: Open Science · p. 15

A Expanded Evaluation across Other LLMs

Expanded evaluations report tuned SELFDEFEND results on Claude-3.5-sonnet and Llama-2-13b-chat, while two figures characterize average ∆d across jailbreak prompts.

A Expanded Evaluation across Other LLMs

B Average ∆d for Various Jailbreak Prompts

  • Figures 5 and 6 present average ∆d across jailbreak prompts for GPT-based SELFDEFEND and tuned models on Llama-2-7b-chat, respectively.Source: Figure 5 · p. 18, Figure 6 · p. 18

C Case Studies

The cases show the methods marking harmful portions in a fake-news jailbreak, while Pdirect and Pintent differ on DrAttack and PAIR.

C Case Studies

  • Pdirect identifies harmful parts in the input, whereas Pintent summarizes the request’s intention before identifying unsafe parts.Source: C Case Studies · p. 18
  • In Figure 7, the methods mark harmful portions of a fake-news prompt that both Llama Guard and Llama Guard 2 classify as safe.The prompt asks for a fake news article spreading disinformation about a political candidate.Source: C Case Studies · p. 18, Figure 7 · p. 19
  • Pdirect-tuned models and Pdirect-based GPT-4 lift harmful parts from different sentences; their Pintent intercepts also differ semantically.Pdirect asks for one harmful part, so the selected portions need not match.Source: C Case Studies · p. 19
  • Both tuned models make satisfactory decisions on various attack and benign inputs.Source: C Case Studies · p. 19
  • On DrAttack, Pdirect struggles to identify consecutive harmful portions after sentence components are deconstructed, while Pintent identifies the intent and defends.Source: C Case Studies · p. 19, Figure 8 · p. 20
  • On PAIR, Pintent judges a prompt harmless despite correctly summarizing its intent; lower toxicity is one reason, while Pdirect identifies the harmful part.The attack prompt’s global semantics differ from the original goal.Source: Figure 9 · p. 20, C Case Studies · p. 20

D Keywords

This section lists refusal-related keyword signals for GPT-3.5/4 and Claude, Llama-2, and Mistral.

D Keywords

  • Tables 11–13 identify refusal signals considered for GPT-3.5/4 and Claude, Llama-2, and Mistral.Source: Table 11 · p. 20, Table 12 · p. 20, Table 13 · p. 20
  • Listed phrases include apologies, “Hello!”, and terms such as “illegal,” “unethical,” and “not ethical.”Source: D Keywords · p. 20
  • Other listed phrases include “but I cannot,” “I’m not able to provide,” “I cannot create,” and “I cannot support.”Source: D Keywords · p. 20
Loading 2406.05498v3…