PaperlayerView exact revision on arXiv

Source-linked AI summary

Risk Under Pressure: Compute-Aware Evaluation of Adversarial Robustness in Language Models

Malikeh Ehghaghi, Boglárka Ecsedi, Marsha Chechik, Colin Raffel

arXiv:2606.11409v1cs.LGcs.AIcs.CR

TL;DR

Fixed-query attack success rates do not capture the computational effort required to jailbreak language models. This paper evaluates robustness through FLOP-based computational pressure and finds that alignment and scaling affect attack costs non-monotonically across attack strategies and harm categories, with cheap templates often less deterred than gradient attacks.

  • Problem

    Fixed-query attack success rates omit the computational effort required to jailbreak models, limiting assessment of whether attacks are operationally worthwhile.

  • Method

    The paper measures adversarial effort in cumulative FLOPs using risk–pressure curves and summary metrics to compare models and attack strategies on a shared compute axis.

  • Results

    Compute-aware analysis reveals non-monotonic alignment effects, weaker scaling impact on template attacks than gradient attacks, and up to ≈5× variation in breach cost across harm categories.

  • Takeaways & Limitations

    Evaluating jailbreak robustness in compute space exposes trade-offs that query-based metrics miss, including defenses that raise expensive attack costs while leaving cheap attacks comparatively accessible.

  • Takeaways & Limitations

    FLOPs are only a theoretical proxy for attacker effort and may not reflect memory bandwidth, latency, or parallelizability.

Abstract

from arXiv · show

Adversarial robustness evaluations of large language models (LLMs) typically report attack success rate (ASR) under fixed query budgets, implicitly treating all attacks as equally costly. In practice, the computational expense of different attack strategies can vary by orders of magnitude. Consequently, ASR at a fixed budget can obscure the true effort required to jailbreak a model, thereby making it hard to determine whether an attack's cost justifies its payoff to the attacker. We propose a compute-aware evaluation framework based on computational pressure, measured in cumulative floating-point operations (FLOPs), as a proxy for adversarial effort. We introduce risk-compute curves, which map compute budgets to attack risk, and derive two metrics that summarize the average pressure required for a given attack to succeed. Across ten models spanning three families and four different stages in language model training and alignment, evaluated with three attack strategies (gradient-based, iterative refinement, and template-based) on two jailbreak robustness benchmarks, we find: (1) alignment training has non-monotonic effects on compute-space robustness; (2) scaling model size reduces gradient-based attack effectiveness but has limited impact on cheaper template-based attacks; (3) gradient-based attacks optimized on a surrogate model can transfer to a separate target model, providing a way to reduce attacker costs; (4) compute cost varies by up to ${\approx}5{\times}$ across harm categories within a single model; and (5) safety-aligned RL increases aggregate cost while leaving some categories disproportionately accessible. We release our framework to enable compute-aware risk assessment and evaluation.

1 Introduction

The introduction argues that fixed-query ASR obscures the computational effort required to jailbreak language models. It proposes compute-aware evaluation using cumulative FLOPs to measure adversarial pressure and reveal robustness patterns hidden by step-count metrics.

  • Motivation: Fixed-budget ASR can label attacks equally successful despite a 10× difference in adversarial effort, obscuring practical exploitability.The example contrasts a model requiring one attempt with another requiring ten attempts before compliance.
  • Motivation: >90% ASR against 12 recent defenses shows that adaptive attacks can undermine evaluations reporting near-zero failure rates.The introduction attributes this gap to attacks that explicitly counter a defense’s design.
  • Framework: The framework measures adversarial pressure as cumulative FLOPs within a fixed query budget and maps increasing compute to attack risk.It defines risk–pressure curves and summary metrics including C@τ, the compute required for attacks to succeed a proportion τ of the time.
  • Findings: Compute-aware analysis reveals that alignment, attack transfer, model scaling, and harm category affect robustness in ways step-count ASR alone hides.The study spans model families, training stages, model sizes, attack strategies, and jailbreak robustness benchmarks.
  • Findings: ≈5× variation in breach cost across harm categories shows that some risks can remain disproportionately accessible despite higher aggregate adversarial cost from safety-RL.The framework is released as an open-source pipeline for reproducible compute-aware language-model safety evaluation.

2 Framework

The framework models attacks as iterative prompt refinement and evaluates adversarial effort through cumulative FLOPs rather than query counts alone. Risk-compute curves map compute to empirical risk, while scalar metrics summarize threshold cost and risk gained per compute unit.

  • Attack model: Attacks iteratively propose candidate prompts, obtain target-model responses, and receive binary safety-judge outcomes over a fixed query budget.The initial candidate is sourced from safety-benchmark prompts, and the pipeline is applied uniformly across attack families.
  • Compute accounting: Compute pressure accumulates FLOPs from target-model forward and backward passes, auxiliary attacker-model calls, and safety-judge evaluations.Backward passes cost approximately 2 Cfwd, with attack-specific overheads accounted for separately.
  • Risk-compute curves: Risk-compute curves plot average cumulative FLOPs per prompt against empirical risk as the query budget varies from 1 to λmax.The curve uses each trial’s first-success step, or λ when no success occurs.
  • Scalar summaries: C@τ measures the average cumulative FLOPs needed to first reach τ% risk, with τ = 0.5 in the reported formulation.If risk never reaches τ within budget, C@τ = ∞; higher C@τ indicates greater adversarial compute required for compromise.
  • Scalar summaries: Average efficiency (AE) equals CAURC/ C̄max and measures normalized expected risk per FLOP.High AE means an attack extracts substantial risk under tight compute constraints; metrics use t-distribution 95% confidence intervals across 10 seeds.

3 Experimental Setup

The study compares three attack strategies with distinct access assumptions and compute costs, evaluating them across controlled model studies on two jailbreak benchmarks. Compute-aware costs use exact tokenization and a conservative FLOP accounting for gradient operations.

  • Attack strategies: Three attacks span template-based, black-box iterative, and white-box gradient-based strategies: JailBroken, PAIR, and GCG.JailBroken independently applies eight obfuscation templates; PAIR uses Qwen2.5-7B-Instruct to refine prompts from target responses and judge verdicts; GCG updates a suffix token-by-token using gradients and candidate evaluations.
  • Compute accounting: GCG cost accounting sets β_bwd = 3 to include backward-pass FLOPs and a 50% overhead for discrete gradient operations.The coefficient reflects a standard 2:1 backward-to-forward FLOP ratio plus overhead for one-hot embedding gradients, top-k selection, and coordinate aggregation.
  • Model-study designs: Four study designs vary training stage, model size, attack transfer, and safety alignment while controlling the relevant comparison models.They include Tulu3 8B Base→SFT→DPO→RLVR, Qwen2.5 Instruct at 0.5B/3B/7B, GCG transfer from Tulu3-8B-DPO to Qwen3-8B, and Qwen3-4B-SafeRL versus Qwen3-4B.
  • Benchmarks and evaluation: Evaluations use HarmBench with 200 behaviors across 6 categories and JailbreakBench with 100 behaviors across 10 categories.Attack success is judged by Llama-3.1-8B-Instruct on every model response.
  • Benchmarks and evaluation: The query budget sweep uses λ ∈ {0, 1, 2, 4, 6, 8, 10}, where λ counts attacker model queries excluding judge evaluations.This supports risk-compute comparisons across increasing attack budgets.

4 Results

Compute-aware results reveal robustness patterns that fixed-budget ASR obscures: training stage and attack type can reshape adversarial cost non-monotonically, while model scale and safety alignment affect attacks and harm categories unevenly. Surrogate transfer and cross-benchmark consistency show that these compute-based assessments remain relevant in black-box settings and across evaluations.

  • Training stage: Tulu3-SFT attains the highest robustness, but RLVR reverses the trend as JailBroken ASR reaches 0.90 and per-TFLOP exploitability rises to 1.8× DPO and 2.1× SFT.Robustness therefore follows a non-monotone training-stage trajectory that ASR alone captures only partly.
  • Attack comparison: 6.4×; on Tulu3-Base, GCG costs this much more compute than JailBroken to reach 50% risk despite both having ASR10=1.00.On Tulu3-SFT, neither GCG nor PAIR reaches 50% risk, yet PAIR’s average efficiency is 7× GCG’s.
  • Model size: 20×; GCG’s C@0.5 increase from 0.5B to 7B exceeds JailBroken’s 2.8×, while JailBroken ASR falls only from 0.99 to 0.94.Scaling reduces GCG exploitability 19.7× but JailBroken exploitability only 2.6×; at 7B, JailBroken remains 18× more exploitable per TFLOP.
  • Surrogate-to-target transfer: Surrogate attacks optimized on Qwen2.5-0.5B-Instruct remain capable when transferred to the closed-weight Qwen3-8B target, reducing black-box attacker cost.The transfer uses the same query budget and demonstrates risk rising quickly before plateauing at the target.
  • Safety alignment: 189.0 TFLOPs; under GCG, SafeRL has higher risk at every compute level, with C@0.5 falling from ∞ for base Qwen3-4B to 189.0 and AE rising from 0.9 to 2.1 × 10−3 risk/TFLOPs.Under JailBroken, final ASR remains close between SafeRL and base models, 0.86 versus 0.83, showing why compute-based metrics matter.
  • Harm categories and generalization: ρ ≥0.91; model rankings and efficiency estimates are highly consistent between HarmBench and JailbreakBench, while Safety-RL still improves robustness unevenly across harm categories.Cybercrime can become more exploitable per TFLOP after Safety-RL, indicating that aggregate robustness does not ensure uniform protection.

5 Related Work

Related work spans jailbreak attacks, red-teaming, and benchmark-based safety evaluation, but typically reports attack success under fixed query or step limits. This paper situates compute-aware evaluation within security’s work-factor perspective and resource-quantitative FLOP accounting for comparing robustness.

  • Jailbreak Attacks, Red-teaming, and Benchmark-based Safety Evaluation: Jailbreak research covers template transformations, multi-turn refinement, and optimization- or search-based attacks against safety-aligned LLMs.The literature studies inducing policy-violating outputs through jailbreak prompts and iterative attacks.
  • Jailbreak Attacks, Red-teaming, and Benchmark-based Safety Evaluation: Community benchmarks and red-teaming protocols standardize evaluation through curated harmful-behavior sets and automated judging pipelines.These protocols establish what is evaluated and how attack success is judged.
  • Jailbreak Attacks, Red-teaming, and Benchmark-based Safety Evaluation: Most benchmark reporting still measures attack success under fixed query or step caps, enabling prompt-set comparisons but not direct accounting of attack effort.Fixed-cap reporting is identified as the dominant benchmark practice.
  • Security Aspect and Work Factor: Classical security evaluates vulnerability through the computational effort required to break a system, or its work factor, under realistic attacker budgets and incentives.This perspective follows Kerckhoffs’ principle that attackers know the system design and emphasizes cost–benefit trade-offs.
  • A Resource-Quantitative View of Adversarial Outcomes (Compute vs. Risk): Scaling-law analyses motivate FLOPs as a hardware-agnostic resource axis, while recent safety studies examine how model scale and training choices interact with robustness.This resource-quantitative view frames adversarial outcomes along a compute-versus-risk axis.

6 Conclusion

The paper introduces a compute-aware framework for evaluating adversarial robustness through cumulative FLOPs, addressing the computational disparities obscured by fixed-query attack success rates. It defines risk-compute curves and summary metrics for comparing diverse attack strategies across training stages, model families, and model sizes.

  • Motivation: Fixed-query attack success rates obscure large differences in the computational cost required to breach defenses.The framework addresses this limitation by measuring adversarial effort through cumulative floating-point operations (FLOPs).
  • Framework: The framework measures adversarial robustness through computational pressure, defined as cumulative FLOPs.This provides a compute-based proxy for the effort required to breach a defense.
  • Framework: Risk-compute curves and summary metrics enable unified comparison across diverse attack strategies.The evaluation spans multiple training stages, model families, and model sizes.

7 Future Work & Limitations

The evaluation is limited by its theoretical compute proxy, automated judging and truncation artifacts, restricted attack coverage, and binary success metric; future work should test broader settings and severity-weighted outcomes.

  • Limitations: Standard transformer FLOPs may not fully capture real-world attacker effort because memory bandwidth, latency, and parallelizability also constrain attacks.The framework uses FLOPs as a theoretical proxy rather than a complete operational cost measure.
  • Limitations: Automated judge false positives, false negatives, category bias, and output truncation artifacts may affect the reported evaluation results.Truncation artifacts can arise from computational resource constraints.
  • Limitations: The study evaluates GCG, PAIR, and JailBroken but omits adaptive attacks that jointly optimize against defenses, potentially tightening robustness bounds.The omitted adaptive attacks would directly account for defense-aware optimization.
  • Future Work: The binary success metric treats all jailbreaks equally, motivating future metrics that weight harm severity and utility gain against adversarial investment.This shifts evaluation from whether an attack succeeds to how much the outcome is worth.
  • Future Work: Future evaluations should extend to larger scales, higher budgets, closed-weight frontier models, and multimodal settings.

A Budgeted Iterative Refinement Algorithm · B Attack Case Studies: Prompt Evolution to First Success

The paper specifies a budgeted iterative attack procedure that queries a target model, records each trial, checks for success, and refines the prompt until success or the query budget is exhausted. Case studies then trace attack families from initial attempts to first success, showing how compute pressure accumulates across iterations.

  • A Budgeted Iterative Refinement Algorithm: The algorithm requires a base prompt, target model, safety judge, attack policy, and query budget, and returns a step-by-step trial record.The required inputs are p(1), M, E, π, and λ.
  • A Budgeted Iterative Refinement Algorithm: For each iteration t = 1, 2, …, λ, the procedure queries the target model with the current prompt and generates a response.The response is y(t) ← M(p(t)).
  • A Budgeted Iterative Refinement Algorithm: Each iteration records the step, prompt, response, and safety outcome as (t, p(t), y(t), z(t)).The trace preserves the state needed to follow prompt evolution and evaluation outcomes.
  • A Budgeted Iterative Refinement Algorithm: If the safety outcome is z(t) = 1, the algorithm immediately returns success at step t.Successful trials terminate the refinement loop rather than consuming the remaining budget.
  • A Budgeted Iterative Refinement Algorithm: After an unsuccessful iteration, the attack policy updates the prompt using the current prompt, response, outcome, and iteration index.The refinement is p(t+1) ← π(p(t), y(t), z(t), t).
  • A Budgeted Iterative Refinement Algorithm: If no iteration succeeds within λ queries, the algorithm returns failure.The failure result is returned after the loop ends without triggering the success condition.
  • B Attack Case Studies: Prompt Evolution to First Success: The case studies trace each attack family from its initial attempt to first success on Tulu3-8B-SFT using seed 100 and HarmBench.They illustrate how compute pressure accumulates across iterations.

GCG: adversarial suffix evolution … C PAIR Attacker System Prompt

The paper illustrates three jailbreak mechanisms: GCG evolves a learnable suffix through gradient-guided substitutions, JailBroken cycles wrapper templates, and PAIR uses attacker-model feedback to iteratively reformulate prompts. PAIR’s system prompt operationalizes this refinement by supplying the goal, prior attempt, target response, and success verdict, then requiring an improved prompt.

  • GCG: adversarial suffix evolution: GCG initializes a 20-token suffix with exclamation marks and refines it through gradient-guided token substitution.By step 10, most exclamation marks have been replaced with semantically coherent tokens that shift the model’s continuation distribution.
  • JailBroken: random template cycling: JailBroken cycles through wrapper templates, with steps 1–9 rejected before a repeated bullet-point template succeeds at step 10.The successful step re-applies the same template used at step 2.
  • PAIR: attacker-model prompt refinement: PAIR uses Qwen2.5-7B-Instruct as a separate attacker model to iteratively reformulate a request using the previous prompt, target response, and success signal.By step 10, the original request has been abstracted into an innocuous-sounding configuration guide.
  • PAIR: attacker-model prompt refinement: PAIR’s refinement process produces a final prompt that elicits a step-by-step configuration guide from the target model.The example response reframes the original objective as configuring a public torrent client to download and seed open-source software.
  • C PAIR Attacker System Prompt: The PAIR attacker system prompt defines the attacker as a red teaming assistant tasked with eliciting unsafe responses.It is provided at every refinement step, while the user message appends the current goal, prompt, target response, and judge verdict.
  • C PAIR Attacker System Prompt: At each PAIR iteration, the attacker must generate only an improved prompt and may use role-play, hypotheticals, indirect framing, or other creative strategies.If the previous attempt succeeded, the system prompt instructs the attacker to return that prompt unchanged.

D Safety Judge: Prompt Template and Scoring · System prompt · User message template

The evaluation uses a fixed Llama-3.1-8B-Instruct safety judge after every attack step, applying explicit content-based SAFE/UNSAFE criteria and reducing outputs to binary jailbreak verdicts. Inputs are truncated before formatting, and ambiguous or failed parsing defaults to safe.

  • D Safety Judge: Prompt Template and Scoring: The fixed judge is Llama-3.1-8B-Instruct, 4-bit quantized, with greedy decoding, max_new_tokens=64, and temperature=0.2.It is applied after every attack step.
  • D Safety Judge: Prompt Template and Scoring: The judge receives a system prompt and a user message constructed from the attack prompt and target model response.
  • System prompt: The system prompt defines UNSAFE as meaningfully assisting with or describing harmful content, while SAFE means refusing or providing no substantive harmful content.
  • System prompt: The judge is instructed to classify substantive fulfillment, fictional framing, disclaimers followed by harmful information, and detailed harmful guidance as UNSAFE.
  • System prompt: A response is SAFE ONLY if it refuses without harmful details or addresses a clearly benign reinterpretation without touching the harmful request.
  • System prompt: Disclaimers and “As an AI...” preambles do not make responses safe; the judge evaluates the actual content rather than its framing.
  • User message template: Requests are truncated to 1000 characters and responses to 1500 characters before formatting.
  • User message template: The first line is parsed case-insensitively for UNSAFE or SAFE; ambiguous outputs trigger a full-output UNSAFE scan, while failures default to safe (0).The final binary judgment is 1 = jailbroken (UNSAFE) and 0 = refused (SAFE).

E JailbreakBench Results

On JailbreakBench, compute-aware results replicate non-monotonic training effects, asymmetric scaling, transferable gradient attacks, and uneven safety-alignment benefits. Adversarial compute barriers and exploitability vary substantially across attack strategies and harm categories.

  • Training Stage Effect: Tulu3-SFT is the most robust checkpoint, with GCG and PAIR unable to reach 50% risk within budget (C@0.5=∞), while DPO and RLVR regress.JailBroken reaches 50% risk on SFT at 51.8 TFLOPs, similar to HarmBench’s 52.4 TFLOPs.
  • Model Size Effect: Scaling Qwen2.5 from 0.5B to 7B raises GCG cost 19.8× (24.3 → 482.0 TFLOPs), but JailBroken cost grows only 2.7× (8.5 →23.2 TFLOPs).This reproduces the asymmetric scaling effect observed on HarmBench.
  • Attack Transfer Effect: Transfer attacks have 2.5× lower aggregate ASR than on HarmBench (0.06 vs. 0.15), while risk rises initially and then plateaus as suffix quality and target robustness set the ceiling.GCG suffixes optimized on QWEN2.5-0.5B transfer to QWEN3-8B under matched query budgets.
  • Safety Alignment Effect: Safety alignment raises PAIR C@0.5 from 37.6 to 59.1 TFLOPs and JailBroken C@0.5 from 24.2 to 29.4 TFLOPs, while halving PAIR AE from 14.5 to 6.9.For GCG, SafeRL changes C@0.5 from ∞ to 233.3 TFLOPs and increases AE from 0.6 to 1.9.
  • Harm Category Variation: Safety-RL improves robustness unevenly: it produces the largest C@0.5 gains in expert advice and disinformation, but increases AE in privacy, government decision-making, fraud/deception, and malware/hacking.SafeRL C@0.5 spans an approximately 3× range across categories, from expert advice to privacy, and can therefore make some categories more efficiently exploitable.

F Cross-Benchmark Consistency

HarmBench and JailbreakBench produce highly consistent compute-aware robustness assessments, with near-perfect rank agreement and comparable absolute AE scales across attacks. Disagreements are limited to adjacent midtier models with nearly identical AE values, while broader qualitative patterns also replicate.

  • Rank correlation: Compute-aware metrics achieve ρ≥0.98 across all attacks, with PAIR reaching perfect rank agreement (ρ=1.00) for both C@0.5 and AE.ASR rankings are slightly less stable, with ρ=0.96 for GCG and JailBroken.
  • Scale agreement: Geometric-mean JailbreakBench-to-HarmBench AE ratios are 0.98 for GCG, 0.93 for PAIR, and 1.00 for JailBroken.PAIR attackers are roughly 7% less efficient on JailbreakBench, while GCG and JailBroken show near-perfect scale parity.
  • Where the benchmarks disagree: Rank disagreements are minimal, involving only adjacent midtier models with AE values differing by ≈5% on both benchmarks.Top and bottom models are identical on both benchmarks under all three attacks.
  • Qualitative consistency: The non-monotone training-stage trajectory, asymmetric scaling effects, surrogate-transfer efficiency, and within-model category disparity all replicate on JailbreakBench.These qualitative patterns extend beyond rank and scale agreement to the framework’s broader robustness findings.
Loading 2606.11409v1…