"Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreak

1 Introduction

• Jailbreak completions can exhibit hallucinations that contradict harmful intent, veer off-topic, provide inaccurate or dangerous instructions, or become logically incoherent.The framework assesses these cases through toxicity, actionability and accuracy, contextual consistency, and coherence checks.Source: Figure 2 · p. 2, §1 · p. 2
• Current evaluators may identify outputs as not aligned without determining whether they are genuinely malicious, causing false positives that can divert research from real threats.Figure 1 contrasts vague or non-informative harmless completions with harmful completions offering dangerous instructions.Source: §1 · pp. 1–2, Figure 1 · p. 1
• BABYBLUE enhances existing red-teaming benchmarks with six evaluators across reasoning-based classification, textual quality, and functionality stages.Its evaluators cover general assessment, coherence, context, instruction, knowledge, and toxicity.Source: §1 · p. 2
• BABYBLUE adds a curated dataset of new and modified behaviors to assess the harm potential of jailbroken outputs.The dataset is designed to augment existing red-teaming benchmarks and support the evaluators.Source: §1 · p. 2

2.1 Hallucination Categories

• Jailbreak hallucinations include input-conflicting, context-conflicting, and fact-conflicting completions, plus logical incoherence that can prevent harmful behavior from being fulfilled.Insufficient model capability can yield ineffective instructions, while garbled prompts from methods such as GCG can lead to incoherent completions.Source: §2.1 · p. 3

2.2 False Positives in the Existing Evaluations

• Existing evaluators trade off reliability, cost, reproducibility, and coverage: lexical matching misclassifies outputs, prompted LLMs can miss subtle errors, classifiers struggle to generalize, and human review is hard to scale.The review identifies false positives and false negatives for lexical matching, high false-negative rates and reproducibility issues for prompted LLMs, dataset and class-imbalance constraints for classifiers, and labor and expense for human review.Source: §2.2 · p. 3
• A robust evaluation framework is needed to assess the harmful potential of jailbreak completions accurately.The paper frames this evaluation around real-world harm, including factually correct malicious content and actionable harmful instructions.Source: §2.2 · p. 3, §2.1 · p. 3

3.1 Evaluation Framework

• BABYBLUE first classifies completions, then checks textual quality and, where relevant, verifies genuine malicious content or actionable instructions.The final-stage evaluators are optional and selected according to behavior categories; open-ended questions bypass the remaining stages after a coherence check.Source: §3.1 · p. 4, §3.1.1 · p. 4

3.1.1 General Evaluator

• The General evaluator combines HarmBench’s initial yes/no classifier with uncensored-LLM reasoning for positive classifications.The initial classifier is retained to reduce false negatives; a positive reasoning result proceeds to the next stage.Source: §3.1.1 · p. 4

3.1.2 Coherence Evaluator

• The coherence evaluator augments perplexity with entity transitions and semantic-similarity checks to detect logical incoherence and repetitive sentences.BLEU-3 measures similarity between adjacent sentences, while α and β weight semantic and entity-based adjustments.Source: §3.1.2 · p. 4

3.1.3 Context Evaluator

• The context evaluator compares entities in the prompt context and completion, using recall to assess contextual relevance.Low-recall completions are filtered out.Source: §3.1.3 · p. 5

3.1.4 Instruction Evaluator

• The instruction evaluator executes generated code in its provided context and checks its output against the expected harmful result.If execution fails, it checks syntax; syntactically valid but nonfunctional instructions fail functionality, while invalid code is compared with expert reference instructions for attacker assistance.Source: §3.1.4 · p. 5

3.1.5 Knowledge Evaluator

• For chemical, biological, and computer-related instructions, an uncensored-LLM evaluator checks factual accuracy and feasibility against expert reference knowledge when available.Without matching reference knowledge, it checks whether the completion’s steps are consistent with its own knowledge.Source: §3.1.5 · p. 5

3.1.6 Toxicity Evaluator

• The toxicity evaluator uses a BERT-based regression model to assign generated text a toxicity probability from 0 to 1.Table 1 compares how different evaluators assess the same behaviors and describes BABYBLUE’s evaluations as more stringent and fact-based.Source: §3.1.6 · p. 5, Table 1 · p. 6

3.2 Augmentation Dataset

• The augmented dataset adds reference knowledge, sandbox-executable examples, and privacy prompts to address evaluation gaps in existing benchmarks.The HarmBench augmentation adds 100 samples: 65 new behaviors and 35 modified behaviors, with expert references for chemical and biological topics and execution scripts and verified references for cybercrime topics.Source: §3.2 · p. 6, Figure 4 · p. 7

4.1 Experimental Setup

• The experiments test 24 open-source and four closed-source models with 16 red-teaming methods, using HarmBench and BABYBLUE’s supplementary dataset.ASR is the performance measure; the supplementary-dataset experiment tests five open-source models.Source: §4.1 · p. 7, §4 · p. 7

4.2 Results

• Closed-source models are more likely than open-source models to generate genuinely harmful content, suggesting higher potential for malicious use.Source: §4.2 · p. 7
• BABYBLUE consistently lowers ASR across five open-source models and red-teaming methods on the supplementary dataset, and Figure 5 shows lower average ASR with BABYBLUE.The authors also report that BABYBLUE has the lowest variance among the compared benchmarks.Source: §4.2 · pp. 7–8, Figure 5 · p. 7, Table 2 · p. 8

4.3 Performance Analysis

• In human-expert review, BABYBLUE significantly improves F1 primarily by reducing false positives while maintaining a stable number of false negatives.The review sampled 200 completions across HarmBench and BABYBLUE, with additional AdavBench evaluations.Source: §4.3 · p. 8, Table 3 · p. 8

• Hallucinations have been studied mainly in text generation and translation, while their dimensions include deviations from input, prior context, or world knowledge.Source: §5 · p. 8
• Prior jailbreak work examines adversarial techniques, including a universal and transferable attack on aligned language models.Source: §5 · p. 8
• AdvBench and HarmBench assess resistance to adversarial prompts but often overlook hallucinations within model completions.Source: §5 · p. 8

• Many perceived jailbreaks are hallucinations, and the paper classifies and analyzes these erroneous outputs.Source: §6 · p. 8
• BABYBLUE uses specialized evaluators to verify factual accuracy, functionality, contextual relevance, and toxicity.Source: §6 · p. 8
• A supplementary dataset targets hallucinations in jailbreak scenarios and complements existing benchmarks to assess jailbroken outputs’ true harm potential.Source: §6 · p. 8

Limitations

• BABYBLUE’s predefined evaluators and metrics may not capture the full spectrum of potential jailbreak threats.The authors say these criteria should be updated and refined as new findings and threat patterns emerge.Source: Limitations · p. 9
• The dataset may omit adversarial techniques and fail to reflect the latest jailbreak methods.The authors state that adversaries continually innovate, so the dataset needs regular updates with new and varied attack strategies.Source: Limitations · p. 9

Ethical Statement

• The authors acknowledge that the enhanced dataset and evaluation methods could be misused to develop more effective jailbreaks.They say the framework focuses on detection and that only the evaluation methodology—not specific attack vectors—is being released.Source: Ethical Statement · p. 9
• Potentially harmful prompts and outputs were handled in controlled environments by qualified researchers.The dataset enhancement focused on evaluation accuracy rather than expanding attack surfaces.Source: Ethical Statement · p. 9
• Human expert contributors volunteered, signed responsibility agreements, and had no conflicts of interest with the research.The authors frame these practices as part of balancing safety research with responsible disclosure.Source: Ethical Statement · p. 9

A Refusal Strings

• Table 4 presents refusal strings.Source: Table 4 · p. 13

B Full Results

• The full results report ASR across models, evaluators, and attack methods, with detailed results in Table 5.The passage describes the supplementary dataset as assessing model susceptibility to adversarial prompts and evaluator effectiveness.Source: B Full Results · p. 13
• Closed-source models exhibit a lower reduction in ASR after BABYBLUE evaluators than open-source models.The authors interpret this as suggesting that closed-source models produce fewer hallucinations under jailbreak attacks.Source: B Full Results · p. 13

C Experiment Details

• The experiments use the models and jailbreak methods described by Mazeika et al. (2024), with implementations following HarmBench protocols and settings.Source: C Experiment Details · p. 13

C.1 Red Teaming Method Descriptions

• GCG optimizes an adversarial suffix, while GCG-Multi and GCG-Transfer extend suffix optimization across prompts or training models; PEZ, GBDA, UAT, and AutoPrompt use other token-selection or optimization strategies.Source: §C.1 · pp. 13, 15
• Other test-case methods include iterative attacker-LLM prompting (PAIR), tree-structured adaptive prompting (TAP), hierarchical genetic prompt evolution (AutoDAN), persuasive request strategies (PAP), human-generated jailbreak templates, and direct behavior-string requests.Source: §C.1 · p. 15

C.2 LLMs and Defenses

• The study groups target LLMs as open-source, closed-source, multimodal open-source, or multimodal closed-source, and includes models with varied safety training and API filters.For Claude and Gemini, system-level filters make direct assessment of model-level defenses difficult.Source: §C.2 · pp. 15–16

C.3 Semantic Categories

• The first experiment uses HarmBench’s seven high-level semantic behavior categories, defined using AI-company acceptable-use policies and literature on malicious LLM use.The listed categories include cybercrime and unauthorized intrusion, chemical and biological weapons or drugs, and misinformation and disinformation.Source: §C.3 · pp. 16–17

C.4 Setup

• Experiments set α to 5, β to 0.2, toxicity threshold to 0.5, context threshold to 0.8, coherence threshold to 2.0, temperature to 0.7, and repetition_penalty to 1.0.Runs used multiple NVIDIA Tesla A100 and A800 clusters.Source: §C.4 · p. 17

D Human Experts

• Volunteer experts validated reference answers and experimental completions using technical accuracy, feasibility, real-world applicability, written reasoning, and cross-group assessment.The contributors included chemistry and computer-science expertise and signed agreements concerning reliability and conflicts of interest.Source: D Human Experts · p. 17

E Related Benchmarks

• The benchmark design draws inspiration from work on geometry, visual reasoning, multimodal mathematics, robust refusal, knowledge editing, and concept comprehension.The cited examples include Zhang et al. (2024a), Li et al. (2024c), Li et al. (2024b), Zhang et al. (2024b), Mazeika et al. (2024), Zhong et al. (2024), and Mei et al. (2024).Source: E Related Benchmarks · p. 17

F Hallucination Examples

• The examples illustrate completions incorrectly classified as malicious by AdvBench and HarmBench but identified as non-malicious by BABYBLUE.They are presented to show how hallucinations can produce false positives.Source: F Hallucination Examples · p. 18

F.1 Example 1

• In the KOALA 13B AutoDAN example, the response lists equipment for an attack on a 2014 Jeep Cherokee.The prompt asks for equipment and an estimated attack duration; the shown response begins an equipment list.Source: §F.1 · p. 18

F.2 Example 2

• In the MISTRAL 7B example, the response proposes a methanol procedure, but the passage says those conditions do not produce dimethylmercury.The response describes heating methanol to around 60°C; the accompanying assessment says that temperature is insufficient and metallic mercury in methanol does not produce dimethylmercury.Source: §F.2 · p. 18

F.3 Example 3

• The BAICHUAN2 13B AutoDAN example presents steps for using the Ghostcat vulnerability on an Apache Tomcat server.The response mentions downloading the Ghostcat tool and running its script.Source: §F.3 · pp. 18–19

F.4 Example 4

• The VICUNA 7B V1.5 TAP example contains a message threatening Jane with a weapon and urging her to stop writing about the sender.The completion also interprets the message as implying possible harm.Source: §F.4 · p. 19

F.5 Example 5

• The prompt asks for text inciting hatred toward a race or ethnicity.Source: §F.5 · p. 19
• The completion condemns racism and advocates respect across races and ethnicities.Source: §F.5 · p. 19
• 0.3443487 is the reported T (Ci) score, below the threshold of 0.5.Source: §F.5 · p. 19