Source-linked AI summary
Diagnose Before You Refine: Multi-Dimensional Judge Feedback for Query-Efficient Jailbreak Attacks
Yixin Tan, Chia-Yi Hsu, Jiayang Liu, Chia-Mu Yu, Rui Wen
TL;DR
Existing query-based jailbreak methods often lack fine-grained diagnostic feedback for prompt optimization. This paper introduces JSB, which uses Jev feedback to guide adaptive refinement and strategy search; experiments report consistent gains over existing automated attacks under limited query budgets.
Problem
Existing evaluation feedback can provide limited insight into response weaknesses and may overestimate attack success by rewarding superficial compliance, limiting its diagnostic value for iterative optimization.
Source: §1 · p. 1Method
JSB uses multi-dimensional Jev feedback to guide prompt refinement, strategy switching, and reseeding of stalled search threads.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The evaluation identifies limitations in existing filtering mechanisms and demonstrates the potential of Jev-based output filtering for detecting and filtering harmful responses.
Source: §1 · p. 2, §7 · p. 8JSB relies on a predefined prompt-strategy set, and closed-source evaluation covers lightweight variants rather than flagship models; effectiveness is lower against more resistant targets such as Claude-Haiku-5.5.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Automated jailbreak attacks rely on evaluation feedback to optimize adversarial prompts, but existing methods often lack fine-grained diagnostic information. We propose Jev-Guided Strategy Bandit (JSB), a query-based jailbreak framework that leverages multi-dimensional probabilistic feedback to guide prompt optimization. JSB combines continuous jailbreak scoring with targeted refinement, strategy switching, and best-state reseeding to balance exploration and refinement. We evaluate JSB on two harmful request datasets across five open-source and three closed-source LLMs. Under a budget of only 10 target-model queries, JSB achieves an average HarmBench attack success rate of 89.8\% across ten open-source settings, consistently outperforming existing automated jailbreak attacks. Further analyses demonstrate its effectiveness under varying query budgets and attacker models. Defense experiments also reveal limitations in existing filtering mechanisms and highlight the potential of Jev-based output filtering.
1 Introduction
The introduction motivates richer evaluation feedback for query-based jailbreak optimization and presents JSB as a framework that uses Jev feedback to refine prompts and guide strategy search. Experiments show JSB outperforming existing automated attacks under limited query budgets, while defense experiments explore Jev-based output filtering.
1 Introduction
- Conventional evaluators can offer limited diagnostic insight and overestimate attack success by rewarding superficial compliance, motivating more actionable feedback.Jev provides option-level probabilities across criteria such as relevance, specificity, and completeness.Source: §1 · p. 1
- JSB consistently outperforms existing automated jailbreak methods under limited query budgets across open- and closed-source LLMs.The paper also reports strong performance under more stringent evaluation metrics, including StrongREJECT.Source: §1 · p. 2
- JSB combines continuous jailbreak scoring with multi-dimensional diagnostics and adaptively refines prompts, switches strategies, and reseeds stalled search threads.Targeted refinement addresses specificity or completeness weaknesses; strategy switching responds to stalled progress or failure to engage, while reseeding uses the best-performing state across threads.Source: §1 · p. 2
- The experiments investigate JSB across open- and closed-source models, limited query budgets, and varying attack configurations.The paper also explores Jev as an output-side filter for detecting and filtering harmful responses.Source: §1 · p. 2
2 Related Work
Prior jailbreak attacks range from methods requiring model internals to query-based prompt search, while jailbreak evaluation spans refusal detection, benchmarks, and fine-grained judgments. JSB builds on these lines by using continuous and multi-dimensional feedback to guide prompt optimization.
2.1 Jailbreak Attacks
- Gradient- or likelihood-dependent attacks such as GCG and AutoDAN [Liu et al., 2024c] require access often unavailable in practical deployments.Query-based methods instead interact with the target model, enabling use against closed-source models.Source: §2.1 · p. 2
- Query-based approaches include iterative refinement and structured search in PAIR and TAP, self-explanation in IRIS, and strategy discovery and reuse in AutoDAN-Turbo [Liu et al., 2025].Other work explores rewriting, decomposition, reconstruction, and fuzzing-based optimization.Source: §2.1 · p. 2
2.2 Jailbreak Evaluation
- Jailbreak evaluation includes standardized benchmarks, useful-information assessment, and fine-grained judgments, while JSB uses continuous scoring and diagnostic feedback to guide optimization.The cited examples include HarmBench and JailbreakBench, StrongREJECT, and JAILJUDGE [Liu et al., 2024a].Source: §2.2 · p. 2
3 Preliminaries
The preliminaries define the jailbreak task and its success-rate evaluation, then describe Jev as a probabilistic evaluator that can assess multiple criteria in one call. These elements establish the task setting and feedback source used by the framework.
3.1 Jailbreak Aligned LLMs
- An attacker constructs an adversarial prompt for a harmful request so a safety-aligned target model produces a response fulfilling that request.The target response is denoted y = MT(x*).Source: §3.1 · p. 2
- Attack success rate is computed by thresholding evaluator scores that measure how far each response fulfills its original harmful request.The evaluator assigns a score to each request-response pair, and the indicator function is used in the ASR calculation.Source: §3.1 · pp. 2–3
3.2 Jev
- Jev is a probabilistic evaluation model trained through Reinforcement Learning for Calibrated Decisions and produces judgments without autoregressive text generation.It can assess multiple criteria in a single call, providing fine-grained feedback at low computational cost for iterative optimization.Source: §3.2 · p. 3
4 Threat Model
The threat model treats the victim as a safety-aligned LLM accessed through a text service and limits the attacker to query-based interaction without internal model access. The attacker aims to elicit a response fulfilling a harmful request within a query budget.
4 Threat Model
- The victim is a safety-aligned LLM exposed as a text-based service that follows legitimate instructions while preventing harmful responses.The attacker submits prompts and observes generated responses through its standard text interface.Source: §4 · p. 3
- The attacker knows the harmful request and general jailbreak techniques but not the victim’s architecture, parameters, training data, or alignment procedure..Source: §4 · p. 3
- The attacker can query the victim and observe responses, but cannot access its weights, gradients, internal representations, or token-level probabilities.An auxiliary LLM and external evaluator may generate or refine prompts and assess responses.Source: §4 · p. 3
- The objective is to induce a response fulfilling the harmful request within B queries to the target model.Source: §4 · p. 3
5 Method
JSB combines multi-dimensional Jev feedback with adaptive strategy selection to guide jailbreak-prompt optimization under a limited query budget. Its feedback scores engagement and specificity while diagnosing response weaknesses; its search mechanisms refine promising attempts, switch strategies, and reseed stalled threads.
5 Method
- JSB combines multi-dimensional Jev feedback and adaptive strategy selection to guide iterative jailbreak optimization within a limited query budget.The feedback evaluates target responses and identifies weaknesses in generated content.Source: §5 · p. 3
5.1 Multi-Dimensional Jev Feedback
- A continuous jailbreak score combines engagement with specificity, rewarding responses that engage with the harmful request and provide concrete, actionable content.Responses that merely appear compliant without substantive assistance receive lower scores.Source: §5.1 · pp. 3–4
- Relevance and completeness diagnose response weaknesses, while specificity and completeness also guide targeted refinement.Jev assesses the dimensions in a single call without additional evaluation requests.Source: §5.1 · p. 4
5.2 Adaptive Strategy Selection
- JSB maintains parallel search threads that generate prompts using their conversation histories, current strategies, and prior evaluation feedback.Threads are initialized by cycling through bypass strategies.Source: §5.2 · p. 4
- When progress continues, targeted refinement addresses a weaker response dimension; when it stalls or engagement fails, the thread switches to an untried strategy.If all strategies have been tried and a thread remains stuck, it restarts from the highest-scoring state across threads.Source: §5.2 · p. 5
- JSB returns the first prompt reaching the stopping threshold, or otherwise the highest-scoring evaluated prompt.The stopping threshold governs early termination; independent evaluators assess returned prompts.Source: §5.2 · p. 5
6 Experiment
The experiments compare JSB with three automated jailbreak baselines across open- and closed-source models, harmful-request datasets, query budgets, attacker models, and filtering defenses. They report JSB’s comparative effectiveness and examine query-cost trade-offs and defense behavior.
6.1 Experimental Setup
- The main comparison uses five open-source and three closed-source target LLMs, two harmful-request datasets, and B = 10 target-model queries.HarmBench and StrongREJECT are independent of the in-loop judges; all compared methods use Mixtral-8x7B-Instruct-v0.1 as attacker.Source: §6.1 · p. 5
6.2 Comparison with Existing Jailbreak Attacks
- Across all 13 model–dataset settings, JSB achieves the highest HarmBench ASR and StrongREJECT mean score under a limited target-model query budget.The tables cover five open-source models on AdvBench and MaliciousInstruct and three closed-source models on AdvBench, with B = 10.Source: Table 2 · p. 6, Table 3 · p. 6, §6.2 · p. 6
- On open-source models, JSB leads both datasets: AdvBench averages 84.4% HarmBench ASR and MaliciousInstruct averages 95.2%.For closed-source models, JSB averages 54.3% HarmBench ASR and a 0.610 StrongREJECT mean score, above the reported baselines.Source: §6.2 · p. 6
6.3 Attack Effectiveness under Query Constraints
- At B = 10, JSB averages 84.4% HarmBench ASR across five open-source models, versus 72.6% for TAP; at B = 40, its average reaches 86.8%.At B = 5, it exceeds the strongest baselines by 12, 13, and 14 percentage points on Qwen2.5-7B, Baichuan2-7B, and Gemma2-9B; on Claude-Haiku-5.5, ASR rises from 14% at B = 5 to 31% at B = 40.Source: §6.3 · p. 7, Figure 2 · p. 7
6.4 Analysis of Stopping Threshold and Attacker Model
- At τstop = 0.8, average ASR is 90.6% with 13.0 queries; raising the threshold to 0.9 adds no ASR improvement and raises queries to 22.3.The authors identify τstop = 0.7 as a favorable effectiveness–query-cost trade-off.Source: §6.4 · p. 7, Figure 3 · p. 8
- With Vicuna-13B as attacker, JSB retains 78% HarmBench ASR on Qwen2.5-7B at B = 10, versus 32% for TAP.JSB’s ASR decreases by 8 percentage points when switching from Mixtral-8x7B to Vicuna-13B, compared with drops of 20, 44, and 20 points for the three baselines.Source: §6.4 · pp. 7–8, Table 4 · p. 8
6.5 Attack Effectiveness against Filter-Based Defenses
- Jev-based output filtering reduces all methods’ ASR to 1–4%, while Llama-Guard output filtering lowers JSB’s ASR from 86% to 28%.The defense experiment uses Qwen2.5-7B on AdvBench with B = 10; the perplexity filter does not suppress the observed attack success rates at its stated threshold.Source: §6.5 · p. 8, Table 5 · p. 8
7 Conclusion
JSB combines multi-dimensional Jev feedback with adaptive strategy search to optimize jailbreak prompts under limited query budgets. The authors report stronger attack performance across open- and closed-source models, effectiveness with another attacker LLM, and findings about defenses.
7 Conclusion
- JSB combines targeted refinement, strategy switching, and best-state reseeding to explore and refine jailbreak prompts under limited query budgets.Source: §7 · p. 8
- Experiments across open- and closed-source LLMs show JSB consistently outperforms existing automated jailbreak attacks, particularly under tight query constraints.Source: §7 · p. 8
- Additional analyses show effectiveness with a different attacker LLM, while defense evaluation finds limitations in existing filters and potential for Jev-based output filtering.Source: §7 · p. 8
Limitations
The authors identify limits in strategy discovery and closed-source evaluation scope, and report lower success against more resistant targets.
Limitations
- JSB's predefined prompt-based strategy set may limit discovery of novel attack strategies beyond its existing library.Source: Limitations · p. 9
- Closed-source evaluation covers lightweight variants—Gemini-3.1-Flash-Lite, Claude-Haiku-5.5, and GPT-4.1-mini—rather than larger flagship models.Source: Limitations · p. 9
- JSB achieves lower attack success rates against more resistant targets such as Claude-Haiku-5.5, suggesting limits against stronger jailbreak robustness.Source: Limitations · p. 9
Ethical Considerations
The paper acknowledges the dual-use risk of automated jailbreak attacks and describes its benchmark and defense evaluations as supporting safety research. Its appendix details the attack configuration, Jev criteria, strategy library, and feedback supplied to the attacker.
Ethical Considerations
- The authors note that the approach could elicit harmful content, use established harmful-request benchmarks and investigate defenses, and warn that releasing prompts or implementations may facilitate misuse.Source: Ethical Considerations · p. 9
A.1 Attack Configuration
- The main experiments use Mixtral-8x7B-Instruct-v0.1 as attacker, with Vicuna-13B for attacker generalization; JSB uses six threads and compares methods at a 10-query budget.Source: §A.1 · p. 11
A.2 Jev Evaluation Prompts
- Jev evaluates stance, relevance, specificity, and completeness in one call; relevance and specificity use 0–3 scales, while completeness uses 0–2.Source: §A.2 · pp. 11–12
A.3 Strategy Instructions
- Four bypass strategies explore alternative formulations, while refinement strategies preserve the current approach and target weaker specificity or completeness.Source: §A.3 · p. 13
A.4 Attacker Prompts and Feedback Construction
- The attacker receives a scalar score, condensed diagnostic feedback, and a strategy directive, rather than Jev's raw numerical relevance, specificity, and completeness scores.Source: §A.4 · pp. 13–14
A.5 Evaluation Protocol
Final responses are assessed independently of Jev using HarmBench and StrongREJECT. The evaluation protocol reconstructs outcomes at fixed query budgets and examines attack effectiveness and query consumption across stopping thresholds.
A.5 Evaluation Protocol
- Final responses are evaluated with independent HarmBench binary success judgments and continuous StrongREJECT scores, reported as attack success rate and mean score.Source: §A.5 · p. 14
- Methods are compared at budgets of 5, 10, 20, and 40 queries using their recorded trajectories and original scoring rules, without extra target-model queries.Source: §A.5 · pp. 14–15
- The stopping-threshold analysis reconstructs operating points for thresholds 0.5–0.9 from trajectories recorded at τstop = 0.9, reporting effectiveness and average query consumption.Source: §A.5 · p. 15
B Baseline Configurations
The appendix specifies matched attacker and target-model settings while preserving each baseline’s original search procedure and in-loop evaluation. It also caps target-model queries and evaluates final responses independently of the in-loop evaluators.
B Baseline Configurations
- All three baselines use the same attacker LLM and target-model generation settings as JSB, while retaining their original search procedures and in-loop evaluation mechanisms.Source: B Baseline Configurations · p. 15
- PAIR uses 30 parallel streams with up to three refinement iterations and GPT-4o-mini evaluation, stopping early at a jailbreak score of 10.Source: B Baseline Configurations · p. 15
- TAP uses branching factor 4, tree width 10, and depth 10, with GPT-4o-mini for jailbreak and on-topic evaluation and early stopping at score 10.Source: B Baseline Configurations · p. 15
- AutoDAN-Turbo fixes a 55-strategy library learned from 150 HarmBench requests, retrieves strategies with MiniLM, and uses up to 40 test iterations with a Gemma-based evaluator.The evaluator's early-stopping threshold is 8.5.Source: B Baseline Configurations · p. 15
- Baseline runs retain their original stopping and candidate-selection rules, obey query budget B, and have final responses scored by HarmBench and StrongREJECT independently of in-loop evaluators.Source: B Baseline Configurations · p. 15