Single-pass Detection of Jailbreaking Input in Large Language Models

1 Introduction

• Jailbreaking modifies harmful prompts to trick a model into producing harmful instructions despite its safety guardrails.The introduction gives an adversarial suffix appended to a bomb-making request as an example.Source: §1 · p. 1
• Existing defenses often require multiple forward passes or auxiliary LLMs, increasing inference time or requiring simultaneous access to multiple models.Perturbation-based methods generate multiple responses and use a majority decision; auxiliary-LLM methods use another model to judge prompt safety.Source: §1 · p. 2
• SPD uses output-token logits to predict harmful content and detect jailbreaking attacks in a single forward pass, without an auxiliary LLM.The approach is motivated by observed differences in output-logit distributions for benign and attacked inputs.Source: §1 · p. 2
• SPD achieves high efficiency and detection rates against unsafe sentences compared with existing approaches in evaluations on open-source LLMs.The reported models include Llama 2, Llama 3, and Vicuna.Source: §1 · p. 2
• SPD is also evaluated on GPT-3.5, GPT-4, and GPT-4o-mini without access to complete model logits.The contribution statement presents this as evidence that SPD can remain promising with partial logit access.Source: §1 · p. 2

2 Related work

• Alignment methods have guarded against malicious outputs for natural inputs but not adversarial inputs.The section frames jailbreaking as an emerging attack category alongside traditional adversarial attacks in vision and language.Source: §2 · pp. 2–3
• Token-level attacks alter and optimize part of the input, while GCG uses a suffix optimized against a target output phrase to elicit harmful content.GCG is described as a universal and transferable attack.Source: §2 · p. 3
• Prompt-level attacks change the whole prompt through techniques such as prefix injection, refusal suppression, role-playing, multilingual prompts, persuasion, and chain-of-thought reasoning.The section also notes automatic and manually crafted ways to generate these prompts.Source: §2 · p. 3
• Post-training detection approaches include perplexity thresholds, LLM-based harmful-content detection, and prompt transformations, though some are computationally expensive or perform poorly on prompt-level attacks.LLM-based detection can use self-examination or another LLM; other approaches include paraphrasing, retokenization, semantic smoothing, prompt optimization, and goal prioritization.Source: §2 · p. 3
• Perturbation defenses use repeated input changes and majority voting, but require many forward passes and can be time- and resource-consuming.The section notes that small perturbations can reduce the success rate of many attacks, especially token-level attacks such as GCG.Source: §2 · pp. 3–4

3 Method

• SPD detects jailbreaking in one forward pass using probabilities of the first few output tokens, without relying on another LLM.The method compares output-token distributions to identify potentially harmful responses.Source: §3 · p. 4

3.1 Notations and preliminaries

• The notation defines x1,n as the input sequence, o_n+1,m as its response, and model logits as the basis for output-token probabilities.The model estimates response probability from the input and preceding response tokens.Source: §3.1 · p. 4

3.2 Motivation

• Because attacks target specified malicious outputs and some evade perplexity defenses, the authors ask whether attacked and benign inputs yield different output-token distributions.They note that attacks may also be iteratively refined with an auxiliary LLM.Source: §3.2 · p. 5
• Attacked inputs show a negative shift in output logits and higher entropy among the first-token probabilities than benign inputs.Benign prompts usually have one or two high-probability first-token candidates, while attacked prompts have more.Source: §3.2 · p. 5

3.3 Single-pass detection

• SPD builds feature matrix H from the top 50 logits at five token positions, while calculating probabilities over the full vocabulary.The authors select five positions because input influence is higher at smaller positions.Source: §3.3 · p. 5
• Table 1 reports dataset sizes by model; attack datasets are sampled at 100% attack success, with no training-test overlap within a model.The same test data are used to evaluate all baselines within each model.Source: Table 1 · p. 6
• A t-SNE plot of 1,000 benign and attacked examples shows a clear class distinction, and the authors use an RBF-kernel SVM to classify H.The classifier labels inputs benign or attacked; attacked inputs should not receive a response.Source: §3.3 · p. 6

4.1 Experimental settings

• The experiments use four jailbreak datasets and two benign datasets, and check generated attack success with Llama Guard.Reported metrics include TP, FP, F1, forward-pass count, and average time; the attack datasets are constructed with 100% attack success.Source: §4.1 · pp. 6–7

4.2 Results on Llama 2 and Vicuna

• SPD outperforms all baselines, reaching 99.8% TP on Llama 2 and 93.7% TP on Vicuna.The comparison covers average forward passes, runtime, TP, FP, and F1 across both models.Source: §4.2 · p. 7, Table 2 · p. 8
• With Llama 2, SPD has less than 1% FP on both benign datasets and almost achieves an F1 score of 1.The evaluation uses AlpacaEval and QNLI as benign datasets.Source: §4.2 · pp. 7–8, §4.1 · p. 7
• SPD uses one forward pass, takes less than 0.4 seconds per input, and is 80× faster than SmoothLLM and 12× faster than RA-LLM.The authors report better performance alongside these speed comparisons.Source: §4.2 · p. 8

4.3 Results on GPT Models

• With top-5 logits, modified SPD detects 87% of attacks for GPT-3.5, 85% for GPT-4, and 91% for GPT-4o-mini.The top-5 setting uses k = 5 and r = 25; the authors report a slight performance decrease when full logits are unavailable.Source: §4.3 · pp. 8–9, Table 3 · p. 9
• On GPT-3.5, GPT-4, and GPT-4o-mini, SPD outperforms the baselines despite limited logit access and few samples.The comparison reports TP, FP, and F1 rates.Source: §4.3 · p. 9, Table 4 · p. 10

5 Conclusion and future directions

• SPD uses an SVM on the first five tokens’ negative log probabilities, runs one pass, is 3× faster than its closest competitor, and exceeds 93% overall TP while keeping FP below 12%.The conclusion attributes the observed attack signal to lower negative log probabilities for attacked-sentence tokens.Source: §5 · p. 10
• With slight modifications, SPD can defend proprietary models without full token probabilities, while full token probability access may greatly improve performance.The conclusion identifies proprietary models such as GPT-4 as a setting where access to complete probabilities is constrained.Source: §5 · p. 10
• The approach requires next-token logits and successful-attack training samples; the authors say few samples can nevertheless train powerful defenses.The conclusion notes that reliance on next-token logits particularly constrains performance for proprietary models such as GPT-4.Source: §5 · p. 10

Broader impact statement

• The authors say low-overhead detection could facilitate faster integration into LLM APIs, but publishing the findings could also help attackers devise ways around the defense.They also describe the method as improving attack detection while maintaining a low false-positive rate for benign inputs.Source: Broader impact statement · p. 10

Contents of the Appendix

• The appendices cover logit-shift analysis, experimental and dataset details, Llama 3 comparisons, ablations, adaptive attacks, ROC curves, and related work on fine-tuning-based attacks.Ablations include unseen-data performance, refusal classifiers, hidden-state features, hyperparameters, and additional prompting on benign samples.Source: Contents of the Appendix · p. 18

A Motivation continued

• For one-token target attacks, the derivation links gradient descent on target cross-entropy to decreases in other logits and corresponding negative log probabilities, motivating the observed shifts.The authors use this analysis as intuition for investigating logit values.Source: A Motivation continued · p. 18
• The authors explicitly state that this analysis is intuition, not a full proof of the observed logit changes, and that exact per-logit dynamics are out of scope.The derivation focuses on optimization-based attacks.Source: A Motivation continued · p. 18
• The same phenomenon was observed with other attack types, and SPD assumes no prior knowledge of attack data or its generation method.Source: A Motivation continued · p. 18

• Experiments used vLLM for model access, OpenAPI access for GPT models, original implementations of defense methods, and one NVIDIA A100 SXM4 80GB GPU; top-p=0.9 and temperature=0 were unchanged.Source: B Experimental setting · p. 18

• Attack prompts were generated from AdvBench harmful behaviors and JailbreakBench behaviors, which contains 100 unique goals drawn from AdvBench and the Trojan Detection Challenge; AdvBench contains 520 unique goals and targets.Source: C Details on the datasets · p. 18
• GCG data used default attack parameters, multi-behavior suffixes, and transfer attacks, totaling 2,042 sentences, 408 unique behaviors, and 551 unique suffixes.Source: C Details on the datasets · p. 19
• AutoDAN used prefixes with multiple targets and transfer attacks between Vicuna and Llama 2, totaling 1,528 sentences, 475 unique behaviors, and 619 unique prefixes.Source: C Details on the datasets · p. 19
• PAIR produced 404 samples across models and was not tested on Llama 2; PAP yielded 30 samples, with no Llama 2 results because too few samples were available.PAP sample availability was limited because the full persuasion taxonomy was unavailable.Source: C Details on the datasets · p. 19
• The study also used an adaptive-attack dataset and filtered unsuccessful attacks with Llama Guard, including transferred GPT-3.5 and GPT-4 attacks.Source: C Details on the datasets · p. 19
• Benign samples included 800 AlpacaEval, 4,000 QNLI, 2,000 AI2-ARC, and 4,000 CodeXGlue examples, with the first two datasets split into equal test and potential-training sets.Source: C Details on the datasets · p. 19
• Llama 2 and Vicuna shared benign test and potential-training data, while GPT models used 500 QNLI prompts; data were randomly split and expanded across behaviors and suffixes or prefixes for heterogeneity.Table 5 shows deliberately cut Llama 2-Chat 7B training examples; all attacked examples were successful, with origins identified by attack or benign dataset.Source: C Details on the datasets · p. 19, Table 5 · p. 20

D Additional experiments with Llama 3

• The Llama 3 comparison indicates that SPD’s effectiveness applies across models even with limited training data.The table compares SPD against prior methods; the supplied caption identifies TP, FP, and F1 rates as its metrics.Source: D Additional experiments with Llama 3 · p. 21, Table 6 · p. 21
• Table 6 reports TP rates, FP rates, and F1 scores for Llama 3, with the best method on each metric highlighted.SmoothLLM is abbreviated as SM, and a diamond marks defenses that do not require analyzing output text.Source: Table 6 · p. 21

E Ablation studies

• The ablations examine data dependency and unseen data, refusal classifiers, logit features, hyperparameters, benign prompting, Bert-based classification, adaptive attacks, and ROC curves.Source: E Ablation studies · p. 21

E.1 Dependency on training data

• For unseen attacks, training on a similar attack group can support strong detection without the exact attack; different attack types may not generalize.GCG training identifies Adaptive attacks, while AutoDAN alone performs poorly on other attacks and PAIR training classifies PAP samples.Source: §E.1 · p. 22

E.2 Performance on unseen benign data

• SPD remains effective on harder benign tasks; Llama 2 performs well without seeing AI2_ARC or CodeXGLUE during training, while Vicuna improves after adding 200 samples from each.The test sets contained 1,000 AI2_ARC samples and 2,000 CodeXGLUE samples; Table 9 reports the FP rates.Source: §E.2 · p. 23, Table 9 · p. 23

E.3 Combining SPD with final layer representation

• SPD and hidden-layer classification are both effective and comparable on Llama 2, while hidden-layer classification improves results with Vicuna.SPD's representation matrix is smaller (250 versus approximately 4,000–5,000), making it slightly faster in inference and training; the hidden-layer classifier requires open-source model access.Source: §E.3 · p. 23, Table 10 · p. 24

E.4 Llama Guard for refusal

• In SmoothLLM’s JailbreakBench implementation, using Llama Guard instead of a string classifier performs better, but is significantly more expensive, requiring an additional LLM and approximately 10 forward passes; reported iterations include Llama Guard.Source: §E.4 · p. 23

The Choice of r and k

• With Vicuna, k below 20 lowers TP considerably; small k also worsens FP, while increasing r beyond about 5 brings little benefit at large k.The main experiments use r = 5 and k = 50.Source: §E.5 · p. 25, Figure 4 · p. 25

The training dataset size T, Tsafe

• With Vicuna, Tsafe below 20 yields relatively high FP, while TP correlates with T; the experiments set T = Tsafe = 200.T and Tsafe are per-dataset sizes; when Tsafe = 50, the benign dataset contains 100 samples.Source: §E.5 · p. 25

Classifier types

• SVM is the preferred classifier in this setting because it keeps FP low, although KNN and XGBoost achieve higher TP rates.The classifiers use the same feature vector H and training set; Figure 6 compares their confusion matrices.Source: §E.5 · p. 25

E.6 Prompting benign samples

• 11.8% to 0.5%: prompting benign samples reduced the initial FDR when tested with an SVM trained without the additional prompt.96.5% of prompted benign responses began with “Sure, here is”.Source: §E.6 · p. 26
• The added prompt shifts benign first-token negative log-likelihoods positively, opposite to the shift observed with jailbreaking attacks.Figure 7(a) plots the first-token negative log-likelihood distributions for benign inputs with and without the prompt.Source: §E.6 · p. 26, Figure 7 · p. 27
• The t-SNE visualization shows separation among attacked, benign, and prompted benign samples.Figure 7(b) uses the H feature vector; its colors distinguish the three sample categories.Source: §E.6 · p. 26, Figure 7 · p. 27

E.7 Ablation studies on adaptive attacks

• Normal GCG achieved a successful attack at step 25, while adaptive GCG never succeeded in 500 steps across attempts and its losses oscillated after step 200.Normal attacks generally succeeded within 20–300 steps; the authors hypothesize that constrained logit optimization conflicts with generating valid jailbreak instructions.Source: §E.7 · p. 27

E.8 Bert-based binary classifier

• RoBERTa has desirable FP rates but performs poorly on AutoDAN and PAP; the authors attribute this to long inputs and only five PAP training samples.The authors also note that an additional language model could itself be attacked, and that training a binary language classifier is more computationally expensive.Source: §E.8 · p. 28, Table 12 · p. 28

E.9 ROC curves

• ROC analysis reports that SPD significantly outperforms baseline methods; its AUROC is almost 1 with Llama 2 and 0.95 with Vicuna and GPT-4.The authors state that Self-defense is not probabilistic, so its AUROC and fixed-FP-rate comparisons are infeasible.Source: §E.9 · p. 28, Figure 9 · p. 29

F Extended related work

• Fine-tuning can introduce safety risks; described defenses include representation noising (Rosati et al., 2024), Vaccine (Huang et al., 2024c), and lazy safety alignment (Huang et al., 2024b).The related-work discussion frames these approaches around preserving safety during fine-tuning and customization.Source: F Extended related work · p. 29