PaperlayerView exact revision on arXiv

Source-linked AI summary

JailGuard: A Universal Detection Framework for LLM Prompt-based Attacks

Xiaoyu Zhang, Cen Zhang, Tianlin Li, Yihao Huang, Xiaojun Jia, Ming Hu, Jie Zhang, Yang Liu, Shiqing Ma, Chao Shen

arXiv:2312.10766v4cs.CR

TL;DR

Prompt-based attacks can elicit harmful content or redirect LLM systems, while existing detectors generalize poorly across attack methods and modalities. JailGuard mutates text and image inputs and compares variant responses to detect attacks. It achieves the best reported detection accuracy on both modalities, with 86.14% on text inputs.

  • Problem

    Existing prompt-attack detectors often target limited attack methods or types, leaving poor generalization across diverse attacks and modalities.

    Source: §1 · p. 2

  • Method

    JailGuard mutates untrusted inputs into variants and detects attacks from divergence among the target model’s responses.

    Source: §1 · p. 3, §4.1 · p. 10

  • Results

    Across text and image inputs, JailGuard achieves the best detection accuracy, with 86.14% on text inputs.

    Source: §9 · p. 33

  • Takeaways & Limitations

    The authors provide an 11,000-sample dataset covering 15 text and image attack types to promote future security research.

    Source: §1 · p. 4

    A TAP prompt framed as benign smuggling guidance bypasses JailGuard and several other detectors, exposing a concrete blind spot.

    Source: §6.3 · p. 27

Abstract

from arXiv · show

The systems and software powered by Large Language Models (LLMs) and Multi-Modal LLMs (MLLMs) have played a critical role in numerous scenarios. However, current LLM systems are vulnerable to prompt-based attacks, with jailbreaking attacks enabling the LLM system to generate harmful content, while hijacking attacks manipulate the LLM system to perform attacker-desired tasks, underscoring the necessity for detection tools. Unfortunately, existing detecting approaches are usually tailored to specific attacks, resulting in poor generalization in detecting various attacks across different modalities. To address it, we propose JailGuard, a universal detection framework deployed on top of LLM systems for prompt-based attacks across text and image modalities. JailGuard operates on the principle that attacks are inherently less robust than benign ones. Specifically, JailGuard mutates untrusted inputs to generate variants and leverages the discrepancy of the variants' responses on the target model to distinguish attack samples from benign samples. We implement 18 mutators for text and image inputs and design a mutator combination policy to further improve detection generalization. The evaluation on the dataset containing 15 known attack types suggests that JailGuard achieves the best detection accuracy of 86.14%/82.90% on text and image inputs, outperforming state-of-the-art methods by 11.81%-25.73% and 12.20%-21.40%.

1 INTRODUCTION

Prompt-based attacks threaten LLM systems, while existing detectors generalize poorly across attack methods and modalities. JailGuard detects attacks by mutating inputs and comparing variant responses, with strong reported accuracy on both text and image inputs.

Motivation

  • Existing detectors often target limited attack types or modalities, generalize poorly to other methods, and can produce many false positives when combined.The authors describe these limitations as making existing methods impractical for real-world systems facing diverse attacks.Source: §1 · pp. 2–3

Approach

  • JailGuard mutates inputs and flags attacks when variant responses diverge, leveraging the lower robustness of attack inputs than benign queries.It implements 16 random mutators and two semantic-driven targeted mutators, then combines mutators using a policy selected from development-set evidence.Source: §1 · p. 3

Results

  • JailGuard’s default combination policy achieved best detection accuracies of 86.14% on text and 82.90% on image inputs, outperforming state-of-the-art defense methods by 11.81%-25.73% and 12.20%-21.40%, respectively.Source: §1 · p. 4

Contributions

  • The authors construct an 11,000-sample dataset covering 15 jailbreaking and hijacking attacks across image and text inputs.They report experiments on this dataset and open-source the dataset and code.Source: §1 · p. 4

Threats to Validity

  • The evaluation covers 11,000 items and 15 attack methods; unseen attacks may still evade detection, and model-specific hyperparameters require tuning for deployment.The authors recommend tuning selected mutators and combination-policy probabilities for the target LLM system.Source: §1 · p. 4

2 BACKGROUND

The background describes LLM systems as pipelines, distinguishes jailbreaks from hijacking attacks, and reviews attack detectors. It also introduces KL divergence, which JailGuard uses to measure differences between response-similarity distributions.

2.1 LLM System

  • LLM systems process user input, query an LLM, and execute a downstream task; JailGuard is placed before the querying stage.Input processing can add system-specific templates or supplementary details, while downstream tools use responses for application-specific tasks.Source: §2.1 · pp. 4–5

2.2 Prompt-based LLM Attack

  • Jailbreaks bypass safety alignment to elicit toxic content, while injection-based hijacks redirect systems toward attacker-desired tasks and outputs.The dataset includes text and image attacks, including eight collected jailbreaking attacks and five prompt-injection attacks plus two image-injection attacks.Source: §2.2 · pp. 5–7

2.3 LLM Attack Detector

  • Detectors may operate before or after an LLM query, but existing approaches can miss unknown attacks or fail to cover both jailbreaking and hijacking attacks.The paper positions JailGuard as a universal detector and compares it with 12 open-source pre-query and post-query methods.Source: §2.3 · pp. 7–8

2.4 Kullback-Leibler Divergence

  • KL divergence measures the difference between probability distributions, reaching its minimum when the distributions are identical; JailGuard uses it to quantify response-similarity differences.The distributions compared are those of LLM responses.Source: §2.4 · p. 8

3 A CLOSER LOOK AT THE MOTIVATION

Examples across text and image modalities illustrate that existing detectors do not cover all prompt-based attacks. SmoothLLM and Azure Content Detector each miss some hijacking cases.

Motivation cases

  • Existing detectors identify only some of the three demonstrated attacks, which span text and image inputs and can produce attacker-desired harmful content.The text cases attack GPT-3.5, while the image case injects adversarial perturbations into MiniGPT-4.Source: §3 · p. 8

Limits of existing detectors

  • SmoothLLM misses hijacking text attacks without refusal keywords and cannot detect image attacks, while Azure Content Detector misses injection hijacks that contain no harmful prompt content.The examples show that jailbreak detection does not ensure coverage of hijacking attacks.Source: §3 · p. 9

4 SYSTEM DESIGN

JailGuard detects prompt attacks by mutating untrusted text and image inputs, then identifying semantic divergence among the target model’s responses. Its 18 mutators include random and targeted operators, and a combination policy selects mutators to improve coverage across attack types.

4.1 Detection Framework

  • JailGuard mutates untrusted inputs, queries the target LLM with variants, and flags an input as an attack when response divergence exceeds a threshold.If all variants produce refusal responses, it flags the input as an attack regardless of divergence.Source: §4 · p. 9, §4.1 · pp. 10–11

4.2.1 Single Mutator.

  • Six random text mutators perturb characters, words, or sentences; translation rewrites the query while aiming to retain benign instructions and semantics.The operators include replacement, insertion, deletion, punctuation insertion, synonym replacement, and translation.Source: §4.2.1 · p. 12
  • Ten random image mutators use geometric, region-based, or photometric perturbations, including flips, masks, blur, color changes, and posterization.Photometric operators alter pixel values, while region mutators perturb selected image regions.Source: §4.2.1 · pp. 12–13
  • Two semantic-guided text mutators identify high-scoring sentences using word-frequency sums and apply replacement or insertion there at five times the probability used elsewhere.Other characters are modified with probability p; important content is perturbed with probability 5p.Source: §4.2.1 · pp. 13–15
  • Targeted mutators focus stronger perturbations on likely attack content, while random character-level changes remain available elsewhere if attackers manipulate which content is selected.The authors state that this makes targeted mutators approximately equivalent to random replacement and insertion under that adaptive strategy.Source: §4.2.1 · p. 15

4.2.2 Combination Policy.

  • Because individual mutators perform unevenly across attack methods, JailGuard samples from a combination pool whose operators and probabilities are selected using development-set empirical data.The selected text pool is Punctuation Insertion, Targeted Insertion, and Translation; the image pool is Random Rotation, Gaussian Blur, and Random Posterization.Source: §4.2.2 · pp. 15–16

5 DATASET CONSTRUCTION

The dataset covers 15 prompt-based attack types across text and image modalities, with 11,000 attack and benign items. The authors collect and validate attack inputs and sample benign queries from existing instruction datasets.

5 DATASET CONSTRUCTION

  • The dataset covers 15 prompt-based attack types across text and image modalities, totaling 11,000 attack and benign items.The authors describe constructing it to address the lack of a comprehensive prompt-based attack dataset.Source: §5 · p. 16
  • The text portion includes jailbreak and hijacking attacks, with successful attack samples retained after validation.It includes 2,000 verified jailbreak prompts and 2,000 verified injection-based hijacking samples.Source: §5 · pp. 16–17
  • Two co-authors verified jailbreak samples, with Cohen’s Kappa of 0.97; a third co-author moderated disagreements.The authors selected prompts rated 5 for policy-violating responses before manual verification.Source: §5 · p. 17
  • The text dataset includes 6,000 benign queries sampled from instruction datasets and maintains a 1:1.5 attack-to-benign ratio.The benign queries cover common-sense questions, role-playing, and logical reasoning.Source: §5 · p. 17
  • The image dataset includes 200 adversarial-based jailbreak inputs, 200 typographic hijacking inputs, and 600 benign inputs.The attack samples were validated for effectiveness on MiniGPT-4.Source: §5 · p. 18

6 EVALUATION

The evaluation examines JailGuard’s mutation-based detection on text and image inputs, including comparisons with existing defenses and generalization across attack types. It also assesses the contributions of its mutator combination policy and divergence-based detection, and characterizes threshold and variant-budget tradeoffs.

6.1 Setup

  • The evaluation uses GPT-3.5-Turbo-1106 for text inputs and MiniGPT-4 for image inputs, collecting TP, TN, FP, and FN and assessing detection with accuracy, precision, and recall.JailGuard generates N=8 variants for each input; when a baseline provides no explicit detection result, refusal keywords in its response are used to infer detection.Source: §6.1 · p. 19

6.2 RQ1: Effectiveness of Detecting Attack

  • JailGuard’s combination policies achieve 86.14% accuracy on text and 82.09% on image inputs, outperforming state-of-the-art methods by 11.81%–25.73% and 12.20%–21.40%, respectively.JailGuard achieves average accuracy of 79.53% on text datasets and 81.68% on image datasets.Source: §6.2 · p. 23
  • Targeted Insertion and Targeted Replacement reach 84.73% and 82.02% accuracy, improving on their random counterparts by 3.42% and 1.07%, respectively.Targeting high-frequency important content helps disrupt long, complex attack templates; adaptive attacks can reduce targeted-mutator accuracy by up to 6.00%, while random perturbations remain effective.Source: §6.2 · pp. 21–22, Table 5 · p. 23
  • On a 100-sample GPT-4o experiment, a more powerful response model improves each mutator’s attack-detection effect by 2%–13%, but JailGuard recommends using the target system’s model.The paper excludes unaligned models from scope and notes that a less capable model may produce unpredictable responses and more false positives, while GPT-4o costs more than twice as much as GPT-3.5-turbo-1106.Source: §6.2 · p. 23

6.3 RQ2: Effectiveness of Detecting Different Kinds of Attacks

  • Mutator performance varies by attack: character-level changes struggle with long template attacks, while Synonym Replacement reaches 96.84% on TAP and Translation reaches 90.31% on Template and 88.84% on Jailbroken.These word- and sentence-level mutators can disrupt benign semantics, yielding only 72.48% and 79.27% accuracy on benign inputs, respectively.Source: §6.3 · pp. 24–25
  • In the Jailbroken case, targeted insertion interferes with important input sentences, and JailGuard detects the attack by response divergence even when only one variant fails and most responses lack refusal keywords.For the benign prompt mentioning a fake chatbot, all JailGuard mutators classify it as benign without false positives; divergence ranges from 0.002 to 0.013, below the built-in threshold θ=0.02.Source: §6.3 · pp. 25–26
  • The combination policy still misclassifies cases when mixed-mutator variants create high divergence for benign inputs or when subtle attacks appear benign to the target model.An idealized upper bound reaches 95.89% accuracy for the text policy and 98.20% for the image policy; the paper identifies subtle, near-boundary attacks as an unresolved detection challenge.Source: §6.3 · pp. 26–27
  • JailGuard’s combination policy exceeds 70.00% accuracy on ten text attack types and reaches 86.17% on benign inputs, while baseline methods vary substantially across attack types.Jailbreak-focused defenses struggle with hijacking attacks, and perplexity-based and LLM-based detection allow only 5.77% and 62.40% of benign samples to pass.Source: §6.3 · pp. 24, 25, 27

6.4 RQ3: Ablation Study

  • 73.82% accuracy with aggregation plus keyword detection and 72.64% with random selection plus keywords were below JailGuard’s 86.14%; the latter achieved only 40.45% recall.The authors report that keyword detection misses many attacks, particularly hijacking prompt injections.Source: Table 6 · p. 28, §6.4 · p. 28
  • Up to 13.82% accuracy degradation followed replacing JailGuard’s combination policy with individual SmoothLLM mutations.Random policies achieved 79.18% and 82.31% accuracy, 6.96 and 3.83 percentage points below the built-in policy; Random Policy 2 precision fell by 10.57%.Source: §6.4 · pp. 27–28, Table 6 · p. 28
  • Modifying either the built-in mutator combination policy or divergence-based detection degraded performance, potentially allowing over 50% of attacks to evade detection.The ablation results identify both components as significant contributors to effective detection.Source: §6.4 · p. 28

6.5 RQ4: Impact of Threshold 𝜃

  • Raising the threshold generally reduces incorrect blocking of benign samples but increases missed attack samples; balancing these outcomes, JailGuard sets θ=0.02 for text and θ=0.025 for image variants.Image variants usually achieve their highest accuracy at θ=0.01–0.03.Source: §6.5 · p. 29

6.6 RQ5: Impact of Variant Amount

  • 66.25% to 100.00% text recall and 60.00% to 100.00% image recall as N rose from 2 to 32; accuracy typically peaked at N=6–14.JailGuard recommends N=8 by default and N=4–6 when balancing effectiveness with resource constraints; the reported MiniGPT-4 overhead at N=8 was 3.95GB.Source: Figure 12 · p. 30, §6.6 · pp. 30–31

7 RELATED WORK

Related work covers prompt-based and other LLM attacks, as well as defenses ranging from input filtering to randomized smoothing. JailGuard distinguishes itself by detecting attacks through divergence in responses to input variants and reports better experimental detection results than baselines, including SmoothLLM.

7 RELATED WORK

  • The paper focuses on multimodal prompt-based attacks that use prompts as the carrier and require neither fine-tuning nor modification of target LLMs or systems.The related work also discusses automatically generated jailbreak and hijacking prompts, backdoor attacks, and privacy-stealing attacks.Source: §7 · p. 31
  • Prior defenses include substring-based safety filtering and system-prompt self-checking, while SmoothLLM uses refusal keywords to classify and aggregate responses.Substring filtering can impose significant overhead on long prompts; the authors report lacking open-source implementations for Self-defend and several other methods.Source: §7 · pp. 31–32
  • JailGuard detects prompt-based attacks by analyzing divergence patterns in LLM responses to input variants, and its experiments report better detection results than baselines.The framework also introduces semantic-guided mutators and a mutator combination policy.Source: §7 · p. 32
  • Unlike traditional randomized smoothing methods that aggregate outputs from perturbed copies, JailGuard uses response divergence to detect potential attacks.Its mutators also include random methods inspired by earlier work [18] [26] [28].Source: §7 · p. 32

8 DISCUSSION

The discussion compares implementation alternatives and identifies query overhead, false positives, and limited attack coverage as challenges for extending JailGuard.

Alternative Solutions in JailGuard

  • 81.70% average accuracy with en_core_web_md was close to en_core_web_lg at 81.90% and bert-base-uncased at 80.20%.The larger alternatives introduce greater memory and time overhead, motivating the smaller model as the default.Source: §8 · p. 32
  • 80.40% accuracy with KL divergence exceeded the 78.20% achieved with MSE, which was more sensitive to threshold selection.The authors retain KL divergence as the default while noting MSE's computational simplicity and potential for resource-constrained settings.Source: §8 · p. 32

JailGuard Enhancement

  • Even with four variants, JailGuard incurs unavoidable extra LLM queries, while using fewer variants degrades detection results.Reducing query costs without sacrificing accuracy remains future work.Source: §8 · p. 32
  • Heuristic benign instructions are the main source of JailGuard false positives, and seemingly toxic benign prompts can also trigger false positives.The authors identify automatic filtering of these inputs as a potential mitigation.Source: §8 · p. 33
  • JailGuard's detection effect may not persist for unseen attacks, motivating continued dataset updates and evaluation on new methods.The current collection includes 15 prompt-based attack methods.Source: §8 · p. 33

Diverse LLM Attacks Detection

  • The authors propose expanding benchmarks to attacks such as data poisoning and backdoors, whose non-robustness could potentially be detected by JailGuard.They also identify designing defenses that generalize across attack inputs as a future direction.Source: §8 · p. 33

9 CONCLUSION

JailGuard detects jailbreaking and hijacking attacks across text and image inputs, with evaluation on a dataset covering 15 attack types.

9 CONCLUSION

  • 86.14% on text and 82.90% on image inputs were the best detection accuracies, outperforming state-of-the-art methods by 11.81%-25.73% and 12.20%-21.40%, respectively.The evaluation dataset contains 11,000 items spanning 15 jailbreaking and hijacking attacks.Source: §9 · p. 33
Loading 2312.10766v4…