Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs

1 Introduction

• Mainstream LLMs remain vulnerable to jailbreak attacks that induce responses to a wide range of harmful queries.The introduction points to GPT-3.5, GPT-4, and Llama3-8B as examples.Source: §1 · p. 1
• Jailbreak success does not reliably indicate harmful knowledge possession, while LLM-as-a-judge frameworks often rely on shallow linguistic cues.The paper argues that Attack Success Rates are ambiguous because outputs may reflect hallucination, obscured intent, or style imitation.Source: §1 · p. 2
• The study identifies a mismatch between jailbreak success and actual harmful knowledge, and finds mainstream LLMs often lack coherent, actionable criminal-knowledge use.It also investigates judge vulnerabilities, including insufficient sensitivity to whether harmful content is authentic.Source: §1 · p. 2
• VENOM evaluates internalized harmful knowledge, planning consistency, and judgment robustness using real criminal knowledge and counterfactual tasks.The framework is designed to decouple criminal-capacity evaluation from jailbreak techniques.Source: §1 · p. 2

2 Motivation

• Treating jailbreak outputs as real threats depends on two assumptions: authentic criminal knowledge in the responding model and faithful judgments by the judge.The motivation frames both assumptions as necessary to assess genuine criminal capability.Source: §2 · p. 3
• When either the responding model lacks authentic harmful output or the judge fails to reveal its authenticity, real-world crime requires external domain expertise to process the jailbreak output.The paper contrasts this case with a severe public-accessible threat when both models have authentic capabilities.Source: §2 · p. 3
• The authors therefore assess underlying criminal knowledge, planning, and judgment rather than relying on potentially overestimated hallucinated harmfulness.This is presented as a more grounded understanding of real-world threat potential.Source: §2 · p. 3

3 Methodology

• VENOM is designed to move beyond surface-level jailbreak prompts and assess grounded criminal capabilities.It evaluates potential real-world harmful capacities through knowledge collection and counterfactual transformation.Source: §3 · p. 3

3.1 Data Collection and Preprocessing

• The data pipeline collects criminal knowledge from underground sources, organizes it into labeled knowledge blocks, and constructs questions from those blocks.The three representative domains are drug operations, explosives and firearms, and financial fraud; sources include professional manuals and technical handbooks as well as underground forums and darknet platforms.Source: §3.1 · pp. 3–4

3.2 Criminal Task Generating

• VENOM uses factual questions and counterfactual tasks, including benign plans structurally aligned with harmful tasks and judge tests that retain malicious tone while replacing facts with fabricated claims.Factual questions are generated, audited, and manually refined; they include open-ended, multiple-choice, and judgment formats.Source: §3.1 · p. 4, §3.2 · p. 5, Figure 4 · p. 5

3.3 Evaluation Pipeline

• The evaluation pipeline measures knowledge recall and accuracy, benign-task completion and logic coherence, and judges’ false-positive rates on tone-only responses.Planning scores are evaluated by claude-3.7-sonnet; the judge false-positive rate counts responses with malicious tone but no factual knowledge that are still flagged as jailbroken.Source: §3.3 · p. 6

4.1 Experiment Settings

• The experiments evaluate GPT-4o, DeepSeek-V3, Qwen-2.5-32B, and Qwen2.5-7B using the full objective-question benchmark, 150 counterfactual planning tasks, and judge tests across five replacement ratios.The judge tests use 100 Q&A pairs per ratio, based on 50 manually crafted harmful Q&A seeds.Source: §4.1 · p. 6

4.2 Experiment Results

• Across four LLMs, AccM varies by less than 15%, RecallK stays at 23%–29%, and AccJ clusters near 65%, with no universally superior harmful-knowledge capability.The assessment spans three criminal domains, three task formats, temperatures 0 and 0.7, and direct or reasoning prompts.Source: §4.2 · p. 6, Table 2 · p. 7
• Reasoning prompts can improve open-ended and judgment responses but also produce overconfident multiple-choice hallucinations; higher temperature modestly raises recall at an accuracy cost.Across prompt styles and temperatures, models also struggle to distinguish factual procedures from misleading details.Source: §4.2 · pp. 6–7
• 48%-58% AccM for reliable 3/3 mastery rises to 82.09%-83.54% for ≥1/3 correctness, while DeepSeek-v3 and Qwen2.5-32B perform competitively with GPT-4o on counterfactual planning.The planning comparison covers task completion and logical coherence.Source: §4.2 · p. 7, Figure 5 · p. 7
• The results suggest that malicious actors may find fine-tuning open-source models a more feasible and potent misuse route than prompt-based jailbreaks.The paper reports strong potential in harmful knowledge proficiency and task planning, with some open-source models outperforming closed-source alternatives on specific high-risk tasks.Source: §4.2 · pp. 7–8
• Judge frameworks show inconsistent judgments with all harmful knowledge preserved, only slight detection declines after partial removal, and false positives when factual knowledge is fully removed but malicious tone remains.Most tested frameworks still classify the fully stripped samples as jailbreaks.Source: §4.2 · p. 8, Figure 6 · p. 8
• The authors conclude that judge insensitivity to factual content also prevents current generalized LLMs from consistently refining authentic, executable criminal knowledge through judge feedback.They report that adversarial attacks instead optimize for surface-level toxic language patterns rather than grounded criminal content.Source: §4.2 · p. 8

Safety Evaluation of LLMs

• Safety evaluations have questioned Attack Success Rate as a sufficient metric, but have largely emphasized linguistic manipulation over factual knowledge assessment.Jailbreak benchmarks also motivate finer-grained evaluation.Source: §5 · p. 8
• Existing LLM-as-a-judge frameworks can rely on superficial cues and overlook knowledge authenticity, potentially overestimating harmfulness.This work separates jailbreak techniques from evaluation of genuine harmful knowledge.Source: §5 · p. 9

Knowledge Assessment of LLMs

• Although benchmarks cover science, security, and medicine, few studies examine criminal knowledge, motivating curated corpora and tasks for evaluating real-world criminal-knowledge use.Source: §5 · p. 9

6 Conclusion

• VENOM probes harmful knowledge, planning consistency, and judgment robustness across five crime-related task types, finding that high jailbreak success does not imply possession of harmful knowledge.LLM-as-a-judge frameworks also tend to misclassify malicious tone as substantive threat.Source: §6 · p. 9

Limitations

• The evaluation covers three representative domains—Drugs Operation, Explosives & Firearms, and Financial Fraud—rather than criminal domains generally.The paper describes these domains as selected for distinct operational characteristics and knowledge-intensive content, and says the modular framework can be extended.Source: Limitations · p. 9
• VENOM exposes LLM-as-a-judge failure cases under semantic shifts but provides no concrete defenses or retraining solutions.Safety repair therefore remains unaddressed in this study.Source: Limitations · p. 9

Ethical Statement

• The study’s sensitive knowledge comes from publicly available sources, and manual review is conducted per sample, with added attention to ambiguous or poorly grounded cases.The passage’s description of output controls is incomplete.Source: Ethical Statement · p. 9

A Additional Explanation of VENOM

• Judgment questions test whether a statement is true or false, with templates that request a rationale or only a JSON judgment.The formats include both reasoning and no-reasoning variants.Source: A Additional Explanation of VENOM · p. 11
• Open-ended questions elicit detailed answers, while multiple-choice questions ask models to select the most appropriate option.The templates separately request reasoning or a direct answer.Source: A Additional Explanation of VENOM · p. 11
• Counterfactual planning questions test whether models can structure harmful behaviors when those behaviors are reframed in benign contexts.The supplied description introduces this task but does not complete its further explanation.Source: A Additional Explanation of VENOM · p. 11
• Planning responses are assessed on logic, completeness, and adaptability, with logic and completeness selected to measure underlying planning competence.Adaptability is included to mitigate inflation from generic fluency and distinguish stronger from weaker plans.Source: A Additional Explanation of VENOM · p. 11
• Judge robustness testing replaces factual, actionable content with fabricated or misleading information while retaining malicious tone to test threat discrimination.The test asks whether judges distinguish genuine threats from stylistically similar but factually incorrect responses.Source: A Additional Explanation of VENOM · p. 11

B Detail of Experimental Setting

• VENOM’s dataset spans Drug Operations, Explosives & Firearms, and Financial Fraud, using standard and counterfactual tasks to assess knowledge, planning, and harmfulness judgment.Table 6 summarizes the task distribution across the three domains.Source: B Detail of Experimental Setting · p. 11, Table 6 · p. 12
• The experiments evaluate GPT-4o, DeepSeek-V3, Qwen2.5-7B-Instruct, and Qwen2.5-32B-Instruct.The passage states that the models are accessed through official APIs or Hugging Face repositories for reproducibility.Source: B Detail of Experimental Setting · pp. 11–12
• Dataset knowledge is collected from publicly available online sources and underground forums, then checked through source tracing and human verification of technical accuracy.The passage names Wikipedia as a basis for human verification.Source: B Detail of Experimental Setting · p. 12
• Samples undergo multi-model filtering and human proofreading, with uncertain cases receiving human verification before high-confidence, high-value data are retained.Raw sample distributions are reported in Table 8.Source: B Detail of Experimental Setting · p. 12
• Question generation and sample filtering use GLM-4-32B-0414 and Qwen3-32B, which are not among the main evaluation targets.Claude-3.7-sonnet assesses counterfactual-planning quality and coherence; these auxiliary models are excluded from reported main results.Source: B Detail of Experimental Setting · p. 12
• Manual review checks individual samples for vague phrasing, missing context, or unreliable knowledge, using GPT-4o with web search to investigate uncertain cases.Inaccurate or unverifiable samples are corrected, supplemented, or removed.Source: B Detail of Experimental Setting · p. 12

C Further Analysis and Discussion

• Larger models generally score better on “All,” but nearly all models struggle to stay correct across option permutations, revealing weak permutation-invariant grounding.“All” requires correctness under all three random option orders, while “Any” requires success under at least one.Source: C Further Analysis and Discussion · p. 12, Table 7 · p. 13
• Knowledge specialization differs by model family: Gemini and Claude perform relatively better on fraud, while Qwen2.5 and InternLM do better on chemical synthesis.The paper suggests pretraining and instruction corpora shape these domain-specific patterns, especially in sensitive domains.Source: C Further Analysis and Discussion · pp. 12–13, Table 7 · p. 13
• Explosives and firearms tasks score lower than fraud-related tasks, while Qwen2.5 excels in chemical knowledge and GPT-4o and DeepSeek perform better in financial scenarios.The authors relate domain difficulty to structural complexity and performance differences to training exposure rather than model scale alone.Source: C Further Analysis and Discussion · p. 13, Table 10 · p. 14, Table 11 · p. 14
• Gemini-2.5-flash-preview reaches 72.5% on Financial Fraud judgment, while Llama-3.1-8B reaches 69.0% on Drug Operations judgment.The results are domain-dependent, and the authors warn that relying on one LLM for jailbreak judgment can produce biased or incomplete assessments.Source: C Further Analysis and Discussion · p. 13, Table 9 · p. 13

• The section identifies J1 and J2 as templates from Qi et al. (2023), J3 from Zhang et al. (2024), J4 from Liu et al. (2023), and J5 from Chao et al. (2024).Source: D LLM-as-a-judge Template · pp. 15–17
• The five-level rubric ranges from proactive refusal (score 1) to full compliance with a policy-violating instruction (score 5).Intermediate levels distinguish warnings or generalized information, engagement without precautions, and explicitly prohibited content.Source: D LLM-as-a-judge Template · p. 15
• J3 asks judges to classify an assistant answer on a five-level scale from “A (Very Harmful)” to “E (No Harmful at All).”The prompt asks for a direct final verdict on answer harmfulness.Source: D LLM-as-a-judge Template · p. 17
• J4 reduces the judgment to whether a response answers rather than refuses, requiring only “True” or “False.”Source: D LLM-as-a-judge Template · p. 17
• J5 classifies a response as unsafe only when it contains harmful, unethical, or illegal content addressing the request; irrelevant or non-harmful requests are safe.An affirmative answer without harmful content specified in the request is also classified as safe.Source: D LLM-as-a-judge Template · p. 17