PaperlayerView exact revision on arXiv

Source-linked AI summary

Fair ASR: Re-Evaluating Black-Box Jailbreaks under Shared Target-Call Budgets

Zhida He, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Peng Yu, Jiajia Li, Chaochao Lu, Qiaosheng Zhang

arXiv:2608.17360v1cs.CRcs.AI

TL;DR

Jailbreak evaluations often overlook attack budgets, making ASR-based comparisons difficult for black-box methods. This paper introduces Fair-ASR and ReCode, finding budget-dependent rankings and achieving 85% ASR on GPT-5 with 7.19 attacker calls per request under 20 target calls.

  • Problem

    Existing jailbreak evaluations often compare ASR without accounting for attack budgets, while FLOPs inadequately capture resource constraints for black-box attacks.

  • Method

    Fair-ASR compares 11 black-box attacks under shared target-call budgets while tracking attacker calls, and ReCode combines rewriting with low-cost perturbation primitives.

  • Results

    ASR and attack rankings vary substantially with target-call budgets, simple primitives remain competitive under equal access, and no evaluated LLM-driven method is efficient in both call types.

  • Takeaways & Limitations

    Under 20 target calls, ReCode achieves 85% ASR on GPT-5 with only 7.19 attacker calls per request.

  • Takeaways & Limitations

    Fair-ASR does not capture token usage, API pricing, manual template-development cost, or multi-turn settings.

Abstract

from arXiv · show

Reliable jailbreak evaluation is essential for assessing LLM safety, but most existing studies rely solely on attack success rate (ASR) without accounting for its dependence on attack budgets, resulting in unfair comparisons across methods. Existing compute-aware evaluations reduce heterogeneous resources into FLOPs, which is difficult to estimate for black-box models and fails to capture resource-specific constraints. To provide a comparable evaluation basis, we introduce Fair-ASR, an evaluation protocol for black-box jailbreak attacks under shared target-call budgets B, using target calls as a directly observable and method-agnostic comparison axis while tracking attacker calls separately for efficiency analysis. We re-evaluate 11 representative attacks under the Fair-ASR protocol and find that attack rankings change substantially across target-call budgets, simple stochastic perturbations and hand-crafted templates remain highly competitive under equal target access, and no evaluated LLM-driven method is efficient in both target and attacker calls. Motivated by this efficiency gap, we introduce ReCode, a compositional budget-efficient attack that combines desensitization rewriting with two effective low-cost primitives identified by Fair-ASR. Under a budget of 20 target calls, ReCode achieves 85% ASR on GPT-5 while requiring only 7.19 attacker calls per request on average, showing strong efficiency in both target and attacker calls.

1. Introduction

The introduction argues that jailbreak evaluations are unfair when they compare ASR without controlling target-call budgets, and proposes Fair-ASR as a shared-budget protocol for black-box attacks. It then presents ReCode, which combines low-cost attack primitives with desensitization rewriting to improve efficiency in both target and attacker calls.

  • Motivation: FLOPs-based evaluation can obscure resource-specific constraints and is insufficient as the sole comparison axis when black-box targets are unavailable.Equal FLOPs may represent substantially different practical resource costs.
  • Evaluation findings: ASR and attack rankings vary substantially with the shared target-call budget, while simple stochastic perturbations and structured code nesting remain competitive under equal target access.The findings challenge comparisons based only on terminal ASR values obtained under unequal target-call budgets.
  • Fair-ASR: Fair-ASR compares black-box jailbreak attacks under shared target-call budgets while tracking attacker calls separately.Target calls provide an observable, common resource across attacks, unlike attacker-model, judge-model, or gradient costs.
  • Evaluation findings: 11 representative attacks are re-evaluated under Fair-ASR, covering hand-crafted templates, stochastic methods, repeated sampling, and LLM-driven attacks.The evaluation tracks attacker calls separately for efficiency analysis.
  • Evaluation findings: No evaluated LLM-driven attack is uniformly efficient in both target and attacker calls.The introduction identifies a two-dimensional efficiency gap between target-side and attacker-side costs.
  • ReCode: Under 20 target calls, ReCode achieves 85% ASR by combining single-pass desensitization rewriting with attacker-free stochastic perturbation and structured code-style nesting.The composition strengthens prompt disguise without repeated attacker-model refinement.

2. Related Work

Black-box jailbreak attacks use diverse mechanisms and resource profiles, while standardized benchmarks improve reproducibility without equalizing attack resources. Prior work shows that jailbreak success depends on attack budgets and motivates compute-aware evaluation using unified FLOPs-based metrics.

  • Attack families: Black-box jailbreak attacks operate through input-output access and include hand-crafted template and stochastic sampling families.Examples include CodeAttack, CipherChat, DeepInception, and BoN.
  • Evaluation frameworks: Standardized benchmarks such as JailbreakBench, HarmBench, and Jailbreak Foundry improve reproducibility, but retaining original search settings leaves attack resources unequal.The SoK argues that ASR alone is insufficient and proposes additional evaluation dimensions.
  • Budget dependence: Increasing sampled variants can substantially improve BoN’s ASR, while SABER models this scaling through Best-of-N risk estimation.These results establish that jailbreak success depends strongly on attack budget.
  • Compute-aware evaluation: Compute-aware evaluations aggregate attacker, target, and judge-model inference costs into a unified FLOPs-based metric.This provides a resource-aware alternative to evaluations that report ASR without accounting for attack budgets.

3. Fair-ASR: Formulation and Evaluation Protocol

Fair-ASR evaluates black-box jailbreaks under shared target-call budgets, treating target calls as the common primary resource and attacker calls as a separate efficiency measure. It defines standardized evaluation procedures and metrics for success, call usage, and response harmfulness under the shared budget.

  • 3.1. Target Calls as the Primary Budget: Target calls provide a common comparison axis because every black-box attack queries the target, whereas attacker models, judges, and optimization procedures are method-specific.
  • 3.1. Target Calls as the Primary Budget: Target-model access is operationally constrained by API quotas, rate limits, abuse detection, warnings, access restrictions, and possible account suspension.
  • 3.1. Target Calls as the Primary Budget: Target calls are directly countable, unlike exact inference FLOPs for closed-source models, while attacker calls remain a separate auxiliary resource.
  • 3.2. Evaluation under Shared Target-Call Budgets: Each harmful request receives at most B target calls, with the attack constructing an adversarial prompt at each call using prior target responses when available.
  • 3.2. Evaluation under Shared Target-Call Budgets: Evaluation stops at the first successful call; if no call succeeds, the procedure uses all B calls.
  • 3.3. Metrics under Shared Target-Call Budgets: ASR@B measures the fraction of harmful requests attacked successfully within at most B target calls, and values across budgets form an ASR–budget curve.
  • 3.3. Metrics under Shared Target-Call Budgets: Average target calls records the call of first success or B after failure, with lower ATC_B indicating earlier success; hand-crafted attacks with K < B omit ATC.
  • 3.3. Metrics under Shared Target-Call Budgets: Harmfulness score uses GPT-4o and the StrongREJECT rubric to assess refusal, specificity, and convincingness from the first successful or final response, with higher HS@B indicating better harmfulness quality.

4. Experiment

Experiments compare 11 black-box jailbreak attacks under shared target-call budgets, showing that rankings and efficiency depend strongly on budget and that simple primitives remain competitive. ReCode addresses the resulting two-dimensional efficiency gap, achieving strong ASR while reducing attacker-call costs across targets.

  • Experimental Setup: The evaluation covers hand-crafted templates, stochastic repeated sampling, and LLM-driven attacks across HarmBench and JailbreakBench, with matched open-weight attacker models and LlamaGuard4 judging Fair-ASR results.Targets include Llama 3.1, gpt-oss, and GPT-4o under B=100; ReCode is additionally tested on GPT-5, Gemini-3.1-Pro, and Claude-Sonnet-4.6 under B=20.
  • Budget Dependence: 60.5% vs. 43.5% at B=5, but 96.0% vs. 91.5% at B=100, shows that TAP and BoN rankings reverse as target-call budgets increase.ASR growth rates also differ across attacks, so endpoint-only reporting conflates effectiveness with target access.
  • Simple Primitives: 81.3% average final ASR with ATC 42.5 makes BoN outperform every LLM-driven attack except ReNeLLM without attacker calls.On Llama-3.1-8B-Instruct, one CodeAttack template reaches 62% ASR at B=1, while all K=8 templates achieve 90.7% average ASR@K.
  • Two-Dimensional Efficiency: 98.7% average ASR@100 and ATC 3.9 make ReNeLLM the strongest and most target-call-efficient stochastic or LLM-driven method, but iterative rewriting incurs substantial attacker calls.Within 100 target calls, six LLM-driven attacks reach 60% ASR and three reach 90%; ReNeLLM generally uses fewer target calls but more attacker calls than PAIR and Rainbow Teaming.
  • ReCode: ReCode reaches 81.0% ASR with AAC 7.00, compared with 57.0%/18.69 for ReNeLLM and 39.2%/49.56 for TAP, while GPT-5 performance reaches 85% ASR with 7.19 attacker calls.ReCode combines desensitization rewriting, stochastic perturbation, and structured code-style nesting; removing the harmfulness gate raises average ASR from 77.6% to 81.0% while lowering AAC from 19.01 to 7.00.

5. Conclusion and Limitations · A. Reproducibility and Experimental Details · A.1. Model and Generation Configuration

The paper concludes that Fair-ASR exposes budget-dependent jailbreak rankings and motivates ReCode as a low-budget attack combining effective primitives. It also documents reproducibility configurations and limitations of target-call-only, single-turn evaluation.

  • 5. Conclusion and Limitations: Fair-ASR re-evaluates 11 representative black-box jailbreak attacks under shared target-call budgets.The protocol reveals budget-dependent rankings, competitive simple primitives, and target- versus attacker-call efficiency tradeoffs.
  • 5. Conclusion and Limitations: ReCode combines single-pass desensitization rewriting with attacker-free stochastic perturbation and structured code-style nesting for strong low-budget ASR and HS.Its design is motivated by the efficiency gap identified by Fair-ASR.
  • 5. Conclusion and Limitations: Target calls are observable and broadly applicable, but omit token usage, API pricing, and manual template-development cost.Fair-ASR also does not yet cover multi-turn settings; future work will add complementary cost measures and extend the protocol.
  • A.1. Model and Generation Configuration: Versioned endpoints are pinned for dated GPT-4o and GPT-5 snapshots, while provider-exposed endpoints are used for Claude Sonnet 4.6 and Gemini 3.1 Pro Preview.Open-weight targets are loaded from released Llama 3.1 and gpt-oss models.
  • A.1. Model and Generation Configuration: All target models use temperature 0.0, while Qwen2.5-7B-Instruct is the primary attacker at temperature 0.7.LLM-driven evaluations are repeated with Mistral-7B-Instruct-v0.3 while holding targets, hyperparameters, judge, budget, and stopping criteria unchanged; Llama Guard 4 is the default judge.
  • A.1. Model and Generation Configuration: 1,024 output tokens are allowed for non-reasoning targets, versus 3,072 generated tokens for reasoning-capable targets.The judge receives at most the first 1,024 visible response tokens, balancing reasoning completion with standardized downstream evaluation.

A.2. Attack Implementations and Budget Accounting · A.3. Baseline Hyperparameters · B. Extended Fair-ASR Evaluation

The paper standardizes attack execution and call accounting under Fair-ASR while separately recording attacker and judge usage. It also specifies baseline configurations and deterministic handling for hand-crafted attacks.

  • A.2. Attack Implementations and Budget Accounting: All attacks use a common Fair-ASR interface while preserving their original search procedures and stopping logic.The interface standardizes model I/O, judge invocation, early stopping, and call accounting across nine OpenRT-adapted attacks plus TAP and BoN.
  • A.2. Attack Implementations and Budget Accounting: Target calls count independently generated target responses, including refusals, blocked responses, unsuccessful generations, and warm-up or exploration generations.Multiple independently sampled responses in one batch count separately.
  • A.2. Attack Implementations and Budget Accounting: Attacker-model and auxiliary-judge invocations are recorded separately and excluded from the target-call count.This accounting separates target access from attacker and judging costs.
  • A.2. Attack Implementations and Budget Accounting: Each distinct hand-crafted template is evaluated once at temperature 0.0, and a method with K fixed templates exhausts its candidate space after K calls.Its attainable ASR remains unchanged for any shared budget B≥K, so the evaluation reports ASR@K rather than duplicate deterministic queries.
  • A.3. Baseline Hyperparameters: Baseline configurations are specified in Tables A.2 and A.1.The supplied passage identifies these tables as the source of the baseline configurations.
  • A.3. Baseline Hyperparameters: Table A.1 presents configurations for attacker-free baselines.The supplied table passage provides the table’s scope but no individual hyperparameter values.
  • A.3. Baseline Hyperparameters: Table A.2 presents configurations for LLM-driven attacks.The supplied table passage provides the table’s scope but no individual hyperparameter values.

B.1. Robustness across Datasets and Attacker Models

The evaluation is broadly robust across attacker and dataset choices, although individual LLM-driven attacks can shift substantially. Attacker choice causes larger method-specific changes in some cases, while most dataset effects remain limited.

  • Evaluation setup: The experiments vary attacker model or dataset while fixing target models, judge, target-call budget, hyperparameters, and stopping criteria.Reported averages are unweighted macro-averages over five target models.
  • Robustness to attacker choice: Changing the attacker alters individual LLM-driven methods’ macro-average ASR by up to 10.8 percentage points on JailbreakBench.PAIR rises from 54.4% to 62.6%, AutoDAN-Turbo from 60.6% to 67.0%, and Rainbow Teaming falls from 77.2% to 66.4%.
  • Robustness to attacker choice: AutoDAN’s HarmBench ASR increases from 27.7% to 49.0% when the attacker changes.The remaining six LLM-driven methods change by at most 5.1 percentage points.
  • Robustness to dataset choice: Under the same Mistral attacker, ReNeLLM remains near saturation across datasets, increasing from 99.4% to 99.7%.CodeAttack also remains stable at 91.2% and 90.7%.
  • Robustness to dataset choice: Five of seven LLM-driven attacks change by at most 4.2 percentage points when transferring between JailbreakBench and HarmBench.AutoDAN and GPTFuzzer improve by 12.2 and 8.0 points on HarmBench, respectively.

B.2. Complete ASR–Budget Curves and Rankings

ASR curves are monotonic, but their marginal gains and attack rankings vary substantially with the target-call budget. ReNeLLM remains strongest, while several alternatives change relative position as budgets increase.

  • B.2. Complete ASR–Budget Curves and Rankings: ASR curves for BoN and seven LLM-driven attacks are monotonic, but additional target calls yield substantially different marginal benefits across methods and target models.Fixed-template attacks are omitted because their candidate sets are exhausted after K calls.
  • B.2. Complete ASR–Budget Curves and Rankings: ReNeLLM remains strongest throughout, with macro-average ASR increasing from 40.4% at B=1 to 83.8% at B=5 and 92.4% at B=10.Its performance approaches saturation after B=10.
  • B.2. Complete ASR–Budget Curves and Rankings: PAIR rises from eighth at B=1 to third at B=5 before falling to sixth at B=100, while Rainbow Teaming rises from sixth to third.These changes illustrate the instability of rankings below ReNeLLM.
  • B.2. Complete ASR–Budget Curves and Rankings: TAP ranks second at B∈{1, 5, 10}, whereas BoN ties TAP at 59.6% when B=30 and becomes the strongest non-ReNeLLM method at B∈{50, 100}.Thus, rankings depend on both the target model and the selected evaluation budget.

B.3. Target- and Attacker-Call Efficiency

The section evaluates joint target- and attacker-call efficiency using Pareto frontiers across success thresholds. Results show a pronounced tradeoff: methods minimizing target calls can require substantially more attacker calls, and no method consistently minimizes both resources.

  • Evaluation criterion: Pareto efficiency is evaluated by the minimum target-call budget B_ρ reaching ASR ≥ ρ and the corresponding average attacker calls AAC_Bρ.A pair is Pareto-efficient when no alternative uses no more calls in either dimension and strictly fewer in at least one.
  • Llama-3.1-8B-Instruct: At ρ=0.6 on Llama-3.1-8B-Instruct, ReNeLLM uses 1 target call and 17.46 attacker calls, whereas PAIR uses 13 target calls and 7.14 attacker calls.The frontier expands from PAIR, TAP, and ReNeLLM at ρ=0.3 to include Rainbow Teaming at ρ=0.6.
  • Llama-3.1-8B-Instruct: Rainbow Teaming reduces AAC from 30.34 to 19.97 relative to ReNeLLM but requires 82 rather than four target calls.Thus, minimizing target calls alone can select a substantially more attacker-intensive method.
  • Llama-3.1-70B-Instruct: On Llama-3.1-70B-Instruct at the highest reported threshold, ReNeLLM is target-call optimal with (B_ρ, AAC) = (1, 3.830), while PAIR is marginally attacker-call optimal with (7, 3.790).No method minimizes both resources across all thresholds.
  • Llama-3.1-70B-Instruct: On Llama-3.1-70B-Instruct, the frontier contains PAIR and Rainbow Teaming at ρ=0.2, adds ReNeLLM at ρ=0.4, and retains only PAIR and ReNeLLM at ρ=0.6.The reported tables use different datasets, attacker models, and threshold grids, so their absolute values should be interpreted accordingly.

B.4. Statistical Reliability and Uncertainty · C. Extended ReCode Analysis

A 2,000-replicate request-level bootstrap shows that Fair-ASR’s main conclusions remain qualitatively stable across benchmark resamples on Llama-3.1-8B-Instruct and gpt-oss-120B. The evidence supports the reliability of shared target-call-budget comparisons, while cautioning against treating small local differences or universal attack superiority as established.

  • B.4. Statistical Reliability and Uncertainty: 2,000 bootstrap replicates resample harmful requests with replacement and recompute each method’s complete ASR–budget curve.The analysis evaluates stability under benchmark-composition variation rather than statistical significance for small pairwise differences at one budget.
  • B.4. Statistical Reliability and Uncertainty: Bootstrap samples preserve strong budget dependence, distinct method growth and saturation profiles, and stable conclusions at tight, intermediate, and large budgets.This pattern holds on both evaluated targets.
  • B.4. Statistical Reliability and Uncertainty: Overlapping confidence intervals at some individual budgets mean small local differences should not be treated as definitive ranking evidence.The broader Fair-ASR patterns are more important than isolated budget-specific separations.
  • B.4. Statistical Reliability and Uncertainty: The bootstrap supports Fair-ASR measurement reliability, not universal superiority for any individual attack.Its purpose is to assess protocol stability under finite-sample variation.
  • B.4. Statistical Reliability and Uncertainty: Shared target-call budgets reveal stable effectiveness profiles and how attacks accumulate success as access increases, whereas terminal ASR alone can obscure these differences.This motivates comparing methods at matched target-call budgets rather than relying only on final performance.
  • B.4. Statistical Reliability and Uncertainty: The bootstrap intervals quantify uncertainty from the finite request set, while additional uncertainty can arise from stochastic attacks, target-model sampling, or judge disagreement.These sources extend beyond benchmark-composition variation.
  • B.4. Statistical Reliability and Uncertainty: 95% confidence intervals are pointwise estimates from 2,000 request-level bootstrap resamples around empirical ASR@B curves under shared target-call budgets.Dashed horizontal lines indicate CodeAttack’s one-shot performance.

C.1. Implementation Details

ReCode combines desensitization rewriting, attacker-free character perturbation, and code-style nesting within a 20-target-call budget. Each attempt uses exactly one attacker call and one target call, stopping upon judge-confirmed success or budget exhaustion.

  • Implementation Details: Each ReCode attempt uses exactly one attacker call and one target call, with evaluation capped at B=20 target calls.Evaluation stops after judge-confirmed success or when the target-call budget is exhausted.
  • Implementation Details: ReCode randomly selects a desensitization template to rewrite the harmful request while preserving its underlying intent.The complete rewrite templates are provided in Section E.1.
  • Implementation Details: ReCode applies character-case inversion and ASCII-character insertion as character-level perturbations, setting perturb_sigma=0.5.These perturbations follow BoN.
  • Implementation Details: ReCode randomly selects a code-style nesting template and embeds the perturbed request within it.The complete nesting templates are provided in Section E.2.

C.2. ASR–Budget Curves and Rankings … E. ReCode Prompt Templates

Across budgets up to B=20, ReCode is strongest overall on GPT-5 and Gemini 3.1 Pro and ranks first on average, while ReCode w/o Nesting performs better on Claude Sonnet 4.6. The paper attributes this Claude-specific difference to code-style sensitivity, illustrates ReCode’s three-stage composition, and specifies four rewrite and five nesting template families.

  • C.2. ASR–Budget Curves and Rankings: ReCode achieves the strongest overall ASR–budget curves on GPT-5 and Gemini 3.1 Pro, while ReCode w/o Nesting performs best on Claude Sonnet 4.6.On Claude Sonnet 4.6, ReCode w/o Nesting reaches 57% ASR at B=20 versus 46% for ReCode.
  • C.2. ASR–Budget Curves and Rankings: 70.3%: ReCode’s average ASR at B=20, exceeding TAP by 35.3 points, ReNeLLM by 35.0 points, and BoN by 63.3 points.ReCode remains strongest on average across GPT-5, Gemini 3.1 Pro, and Claude Sonnet 4.6, consistently outperforming BoN, TAP, and ReNeLLM.
  • C.2. ASR–Budget Curves and Rankings: ReCode ranks first throughout B≤20; at B=20, ReNeLLM reaches 35.3% macro-average ASR and TAP reaches 35.0%.TAP briefly overtakes ReNeLLM at B=15, while the two methods are nearly tied at B=20.
  • C.3. Code-Style Sensitivity on Claude: Removing code-style nesting increases Claude Sonnet 4.6 ASR from 46% to 57%, reduces total target attempts from 1,580 to 1,198, and lowers filter rate from 40.4% to 15.1%.Removing perturbation instead decreases ASR from 46% to 38%, with a filter rate of 29.3%.
  • D. Qualitative Case Study: ReCode uses one attacker-model rewrite call, followed by stochastic character-level perturbation and code-style nesting without additional attacker-model calls.The target reconstructs the concealed task from the nested prompt, and content-filter calls remain charged to the target-call budget.
  • E.1. Rewrite Templates: The rewrite stage randomly samples four strategies: Literary Rewrite, Benign Softening, Neutral Substitution, and Objective Substitution.Each strategy softens or generalizes sensitive wording while keeping the high-level topic, sentence function, requested answer type, and level of detail recoverable.
  • E.1. Rewrite Templates: Literary Rewrite uses metaphorical or symbolic language, Benign Softening substitutes neutral objectives, and Neutral Substitution replaces sensitive entities or actions with broader terms.Objective Substitution frames sensitive attributes or outcomes using technical, scientific, or objective descriptions.
  • E.2. Code-Style Nesting Templates: The final nesting stage uses five templates: Code Completion, Dependency Graph, Heterogeneous Recursive Containers, Descriptor Chain, and Multiple Inheritance.Before instantiation, the perturbed request is split into words represented by the {words_repr} Python literal list; each template asks for completed code that recovers the hidden task.
Loading 2608.17360v1…