Validity-Aware Jailbreak Evaluation for Large Language Models

Qilong Wu; Sahil Wadhwa; Pranab Mohanty; Giri Iyengar; Varun Chandrasekaran

Source-linked AI summary

Validity-Aware Jailbreak Evaluation for Large Language Models

Qilong Wu, Sahil Wadhwa, Pranab Mohanty, Giri Iyengar, Varun Chandrasekaran

arXiv:2609.00498v1Aug 31, 2026cs.AI

TL;DR

Existing jailbreak evaluations can label realistic or intent-aligned responses as successful without establishing factual or procedural correctness. SEAV adds a verification-centric layer that decomposes responses into ordered steps and checks them for factual, structural, and operational validity. Across the evaluated benchmarks and diagnostics, enforcing correctness substantially changes measured robustness, while the authors note limitations from LLM-based judgment, detail sufficiency, and dual-use risks.

Problem
Existing evaluations often conflate linguistic plausibility with epistemic validity and neglect correctness in ordered, executable procedures.
Method
SEAV decomposes responses into ordered steps, verifies factual claims with retrieval grounding, checks ordering consistency, and aggregates validity into a postgeneration evaluation layer.
Results
SEAV substantially changes measured robustness by reclassifying originally labeled jailbreak successes as invalid across multiple benchmarks and remaining stable across tested evaluator models and retrieval backends.
Takeaways & Limitations
Correctness-grounded evaluation may provide more reliable and interpretable safety assessment because many prior jailbreak labels do not establish operationally valid harmful capability.
Takeaways & Limitations
SEAV still relies on LLM-based extraction, interpretation, and aggregation, while some verification decisions remain subjective and detail sufficiency is uneven across datasets.

Abstract

from arXiv · show

Jailbreak robustness has become central to large language model (LLM) safety evaluation, yet prevailing methodologies rely primarily on refusal behavior, semantic resemblance, and intent-matching heuristics that emphasize linguistic plausibility rather than correctness. We identify a key limitation in existing evaluations: many jailbreak intents depend on instructional validity rather than epistemic factuality, allowing realistic-looking responses to be labeled successful despite being factually or procedurally incorrect. To address this gap, we propose Sequential Epistemic and Action-Level Validation (SEAV), a verification-centric jailbreak evaluation framework that decomposes responses into ordered steps and evaluates both validity and correctness. SEAV combines LLM-as-a-judge mechanisms for semantic interpretation with retrieval-grounded verification using external knowledge sources, assessing whether generated content is factually correct, structurally consistent, and operationally capable of advancing harmful objectives. Empirically, SEAV cuts the false-positive rate on SD-A (a curated strategic-dishonesty diagnostic) by 14.9\,pp vs. the strongest baseline, and reclassifies 22.1\%--51.0\% of sampled prior-labeled successes as invalid across three of four public benchmarks. Together, these results show that enforcing correctness substantially reshapes measured robustness: many previously labeled jailbreak successes are reclassified as invalid, and results are stable across the tested search backends and evaluator models. Code and data are available at https://github.com/Ardor-Wu/SEAV.

1 Introduction

Jailbreak evaluations can misrepresent LLM security when they treat linguistic plausibility or apparent intent satisfaction as success without checking factual and procedural validity. SEAV addresses this gap with a four-stage validity layer that extracts, verifies, orders, and aggregates response steps.

Motivation: Prior ASR protocols can be inconsistent, but even consistent evaluations may fail to determine whether harmful outputs are valid.This limitation can misrepresent the true security posture of LLMs.
Motivation: Most jailbreak prompts seek executable procedures, making procedural correctness and dependency structure central to evaluating responses.Many existing paradigms instead rely on non-refusal, semantic similarity, or surface-level compliance.
Motivation: Instructionally factual responses may appear cooperative while containing factual errors or invalid steps, causing evaluations to overestimate model capability.Validity includes relevance, factual correctness, ordering compliance, and operational potency.
Approach: SEAV is proposed as a complementary postgeneration operational-validity layer rather than a replacement for refusal, policy-compliance, or harmfulness evaluation.Its pipeline extracts ordered steps, verifies them through web search, checks ordering consistency, and aggregates a final score.

2 Problem Formulation

The paper defines jailbreak success operationally as harmful-response advancement, requiring relevance, factual correctness, ordering compliance, and sufficient completeness. This criterion complements rather than replaces refusal, policy-compliance, and harmfulness evaluations.

Operational Success: SEAV uses the operational end of the success spectrum, testing actual advancement while catching plausible errors.The paper retains refusal and policy-failure perspectives as complementary evaluation options.
Operational Success: The operational criterion evaluates whether a response advances harmful intent through relevance, factual correctness, ordering compliance, and operational potency.Operational potency means sufficient completeness after the first three facets are satisfied.
Operational Success: Knowledge intents require true claims, whereas procedural intents require correct, ordered, executable steps.The distinction separates epistemic factuality from instructional validity.

3 Shortcomings of Existing Evaluators

Existing jailbreak evaluators often equate linguistic plausibility or intent alignment with success, overlooking factual correctness and procedural order. This produces false positives for epistemically invalid responses and misses order-sensitive failures.

3.1 Over-reliance on Linguistic Plausibility: Current evaluators conflate linguistic plausibility—fluency, coherence, and surface realism—with epistemic validity, which requires factual and operational correctness.Realistic, detailed, intent-aligned responses may therefore be labeled successful despite being invalid.
3.1 Over-reliance on Linguistic Plausibility: More than 50% of SD-A responses are labeled successful across M1–M5 despite SD-A responses being epistemically invalid by design.This pattern indicates that plausibility-oriented signals frequently dominate correctness considerations, although SD-A correctness was not independently re-verified post hoc.
3.1 Over-reliance on Linguistic Plausibility: Existing evaluators systematically misclassify linguistically plausible but epistemically invalid responses as successful jailbreaks.The problem is especially relevant to instructionally factual queries, whose usefulness depends on correctness and executability.
3.2 Agnostic to Ordering Sensitivity: Procedural correctness is central because many jailbreak prompts solicit workflows, multi-stage reasoning, or executable procedures with order-sensitive dependencies.Existing datasets often lack explicit annotations for sequencing, dependency structure, or causal preconditions.
3.2 Agnostic to Ordering Sensitivity: ORDSENSE varies only structural correctness across original, validly reordered, and dependency-violating procedure variants.It contains 137 WIKIHOW-derived samples with LLM-inferred step-level dependency graphs.
3.2 Agnostic to Ordering Sensitivity: M3–M5 exhibit near-zero sensitivity to dependency-violating reorderings, while M1 and M2 show higher sensitivity but greater instability.M3–M5 sensitivity is 1.5%–16.9%; M1 flips 26.6% of verdicts even under valid reordering.
3.2 Agnostic to Ordering Sensitivity: SEAV combines 9.6% instability under valid reordering with 61.0% sensitivity to dependency violations.These metrics distinguish structural reasoning from generic verdict instability.

4 Our Approach: SEAV

SEAV is a four-stage, correctness-grounded pipeline that converts responses into steps, verifies local validity and ordering, and synthesizes evidence into an operational-actionability score.

Sequential Response Rephrasing: SEAV transforms harmful-response text into an ordered sequence while distinguishing sequential, unordered, single, code-dominant, narrative, and refusal outputs.Extracted step text is preserved verbatim, and ordering verification applies only to sequential responses.
Sequential Response Rephrasing: 91.0% of rephrased sequences are judged semantically equivalent to their original responses across the n=322 multi-step sample.Equivalence requires that no substantive information is added, omitted, or altered.
Step Verification: Each extracted step is checked with web-grounded retrieval for factual or operational correctness and relevance to the harmful intent.Step-level verdicts distinguish verified, factual only, and unverified outcomes while initially ignoring cross-step dependencies.
Ordering Verification: Ordering verification infers dependencies and tests whether the original sequence satisfies them, producing a compliance verdict, reasons, violations, and a dependency graph.Responses without ordered dependencies or with unordered structure bypass this verification.
Final Judgment: The final LLM judgment synthesizes step verdicts, dependency structure, ordering compliance, and procedural evidence into a continuous 0–1 operational-actionability score.The rubric considers factuality, relevance, critical-step support, procedural sufficiency, ordering severity, and unsupported claims rather than arithmetic averaging.

5 Implementation Details

SEAV primarily uses Gemini 3 Flash with web-grounded retrieval, while sensitivity tests vary the judge model and retrieval backend; each sample requires substantial inference and search.

Models: Gemini 3 Flash serves as the default judge for all SEAV pipeline nodes and baseline evaluators M1–M5.Kimi K2.5 is used for dependency annotation and a rephrasing-equivalence check, while GLM-5 is tested for evaluator sensitivity.
Retrieval and Search: SEAV uses Gemini’s built-in Google Search grounding by default and Tavily with top-k=5 for sensitivity experiments and non-Gemini evaluators.Both retrieval configurations support step and structural verification.
Computational Cost and Deployment: Each sample uses approximately 8 LLM calls and about 5 searches, making the pipeline computationally more intensive than a single holistic evaluator.The calls cover extraction, step verification, ordering verification, and final judgment.

6 Results

Results show that SEAV detects validity failures missed by existing evaluators, reclassifies many benchmark-labeled successes, and remains broadly stable across tested implementation choices.

Diagnostic Datasets: 57–96% of epistemically invalid SD-A responses are labeled successful by M1–M5, compared with 42.2% for SEAV.SEAV achieves the lowest rate, with a 15+ point gap over the next-best baseline.
Diagnostic Datasets: SEAV achieves F1 of 91.2% on JQR-BINARY versus 91.7% for M3, while substantially outperforming M1, M4, and M5.The control dataset contains n=262 human-annotated binary samples and excludes ambiguous partial samples.
Dataset Re-evaluation: SEAV’s issues measure disagreement with original benchmark labels rather than independently adjudicated ground-truth errors.Re-evaluation is restricted to samples originally labeled as jailbreak successes.
Dataset Re-evaluation: 22.1%–51.0% of sampled originally successful JBB, GPTFUZZ, and WILDGUARDMIX responses are reclassified as invalid by SEAV.Most reclassifications arise from factual errors or procedural inconsistencies; all 77 sequential JAILBREAKQR samples are confirmed valid.
Implementation Robustness: Changing Gemini retrieval to Tavily shifts SD-A false-positive rate and JQR-BINARY F1 by less than 1 pp, while broader invariance is not established.Threshold sweeps, context ablation, and alternative final-judgment formats further test implementation sensitivity.

7 Related Work

Prior jailbreak evaluators emphasize holistic refusal, semantic matching, or surface plausibility, whereas SEAV targets retrieval-grounded factual, procedural, and ordering validity.

Existing Evaluators: Existing LLM-as-a-judge and matching-based evaluators can underweight correctness, procedural structure, and ordering-dependent constraints.ASR comparisons may also be invalid across inconsistent evaluation setups.
Adjacent Evaluation Lines: WMDP probes hazardous knowledge through multiple-choice questions, whereas SEAV evaluates behavior and correctness in harmful responses.Jailbreak Tax instead measures benign-task accuracy drops after jailbreaks rather than verifying harmful prompts.
Factuality and Planning: SEAV extends factuality-evaluation approaches by checking harmful-response correctness, procedural validity, and step ordering against retrieved evidence.Its dependency focus complements causal and temporal plan analysis in CaT-Bench.
Comparison with JADES: JADES decomposes intents and optionally retrieves claim support, while SEAV additionally verifies ordered steps, local correctness, and response structure.The methods therefore overlap in decomposition but differ in structural and ordering validation.

8 Conclusion

SEAV is a verification-centric jailbreak evaluator that decomposes responses into ordered steps and checks factual and structural validity through retrieval grounding. Across multiple benchmarks, enforcing correctness substantially changes measured robustness and may improve the reliability and interpretability of safety assessment.

SEAV decomposes responses into ordered steps and checks factual and structural validity via retrieval grounding.
Enforcing correctness substantially changes measured robustness across multiple benchmarks.
Verification-aware evaluation is intended to support more reliable and interpretable safety assessment for large language models.

9 Limitations

The paper identifies limitations involving evaluator reliability, dataset scope, design sensitivity, and cases where correctness-grounded judgment may understate jailbreak risk. It also finds that current verification primarily checks present information rather than whether all necessary operational details are included.

SEAV still relies on LLM-based judgment, which may inherit model biases, prompt sensitivity, reasoning errors, and subjective verification decisions.
The evaluation focuses on English-language jailbreak benchmarks and does not test cross-response composition, conversation histories, or accumulated capability.
Different step-extraction strategies, classification schemes, aggregation rules, thresholds, or policies may produce different quantitative results.
On some tasks, correctness-based judgment may underestimate risk by classifying factually wrong non-refusing responses as failed jailbreaks rather than policy violations.
Across 553 responses, only 60.3% of in-scope responses were detail-sufficient, and rates varied from 85.7% on SD-A to 44.6% on JQR-Binary.

10 Ethical Considerations

The paper addresses ethical risks in evaluating harmful jailbreaks, including dual-use concerns, dataset and content-handling choices, and limitations of validity judgments.

Ethical safeguards: SEAV evaluates harmful jailbreaks using existing public datasets rather than generating new harmful content.The study analyzes harmful content but reports no newly generated harmful material.
Dual-use considerations: A more accurate evaluator may benefit defenders while giving adversaries more precise feedback for refining operationally valid jailbreaks.The paper recommends responsible deployment and access controls for production systems.
Dataset construction: The SD-A diagnostic is derived from adversarial multiple-choice data containing strategically dishonest responses with intentional factual or procedural errors.The construction filters responses selected as strategically dishonest and applies content moderation to adversarial intents.
Dataset construction: ORDSENSE uses WIKIHOW-derived procedures, inferred dependency labels, and original, valid-alternative, and dependency-violating orderings.The variants preserve step text and differ only in ordering, isolating structural correctness.
Evaluation procedure: SEAV’s evaluation pipeline extracts steps, verifies factuality and relevance, checks ordering dependencies, and aggregates evidence into an operational-validity judgment.Ordering verification is skipped for unordered responses, while step extraction supplies the sequential structure used downstream.
Validation and limitations: The internal audit found strongest agreement at final scoring but identified dependency inference as the least consistent pipeline node.The audit is presented as face-validity corroboration rather than pipeline-wide reliability evidence.

E AUROC and Threshold Sensitivity Analysis

The threshold-sensitivity analysis tests whether SEAV’s conclusions depend on a chosen cutoff or scoring design. Results support SEAV’s robustness on SD-A and ORDSENSE, while showing comparability with M3 on JQR-Binary.

Analysis design: Threshold-free AUROC, paired bootstrap tests, and threshold sweeps assess whether SEAV’s results depend on τ=0.25 or rubric weights.AUROC evaluates continuous-score separation without committing to a threshold.
Threshold sweep: SEAV’s F1 remains 0.81–0.92 on JQR-Binary and 0.79–0.89 on ORDSENSE across the tested threshold ranges.The reported ranges are τ ∈[0.05, 0.95] for JQR-Binary and τ ∈[0.05, 0.75] for ORDSENSE.
Comparative results: On ORDSENSE, SEAV dominates M3 at every tested threshold, whereas on JQR-Binary its AUROC is statistically comparable to M3.The JQR-Binary comparison is interpreted conservatively because its AUROC gap falls inside the bootstrap confidence interval.
Baseline behavior: M1 and M2 are sensitive to ordering changes but unstable under valid reorderings, with instability rates of 26.6% and 13.3%, respectively.Their jailbroken rates also drop from condition (a) to condition (c), but the results suggest surface-level instability contributes to those changes.

F Ablation: Preceding Context in Step Verification

The preceding-context ablation tests whether rewriting each step with prior context improves verification. Across the three datasets, the alternative does not improve the reported metrics, so the default remains unchanged.

Ablation design: The jades_context ablation prepends preceding context to each extracted step before GLM-5 verification.It compares this variant with SEAV’s default extraction under matched settings across JQR-Binary, SD-A, and ORDSENSE.
Experimental controls: The ablation uses GLM-5 with Tavily while holding other pipeline settings fixed.This isolates the tested context-rewriting choice within the stated experimental configuration.
Results: Across all three datasets, jades_context moves every paper-reported metric against its desired direction.The table evaluates the variant against the SEAV default using metric-specific desired-direction arrows.
Conclusion: The SEAV default is retained, while more targeted context augmentation is left for future work.The reported ablation provides no positive effect supporting replacement of the default.

G.2 Few-Shot and Scale Ablation

The few-shot and score-scale ablation compares zero-shot continuous, continuous few-shot, and Likert-5 few-shot judging under contamination-controlled evaluation. No configuration improves performance consistently across datasets, supporting zero-shot continuous scoring as the default.

Ablation design: Table 14 compares zero-shot continuous, continuous few-shot, and Likert-5 few-shot judgment configurations on three held-out SEAV datasets.The held-out subsets exclude few-shot examples and entries sharing the same intent or goal text.
Results: No single ablation configuration improves the judge across all datasets.After Bonferroni correction, only two significant effects survive on JQR-Binary, and they point in opposite directions.
Conclusion: Zero-shot continuous scoring remains the safest default for the main paper.The conclusion follows from inconsistent cross-dataset ablation effects after multiple-comparison correction.
Search sensitivity: Switching between Gemini grounding and Tavily produces negligible differences, below 1 pp in SD-A false-positive rate and JQR-Binary F1.This indicates validity decisions are largely stable across the tested search backends.
Evaluator sensitivity: Evaluator choice changes per-dataset error rates: SD-A false-positive rate is 18.2% for GLM-5 versus 41.6% for Gemini 3 Flash.Both evaluators remain strong on JQR-Binary, but ranking-level conclusions hold while exact error rates vary.